Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MaterializedPostgreSQL fails on data update in case of postgresql generated columns. #40449

Closed
istarkov opened this issue Aug 20, 2022 · 0 comments · Fixed by #57568
Closed
Assignees
Labels
experimental feature Bug in the feature that should not be used in production

Comments

@istarkov
Copy link
Contributor

istarkov commented Aug 20, 2022

Clickhouse version affected 22.3.7.28, 22.8.1.2097
postgres 14

Imaging following table in postgresql.

--postgres
CREATE TABLE test (
    id integer PRIMARY KEY,
    x double precision,
    y double precision GENERATED ALWAYS AS (x*2) STORED
);
insert into test values (1,1);
insert into test values (2,2);

then at clickhouse create

-- clickhouse
CREATE DATABASE IF NOT EXISTS {REPLICATION_IDENTIFIER:Identifier}
ENGINE = MaterializedPostgreSQL('db:5432', database', 'login', 'pass')
SETTINGS materialized_postgresql_tables_list='test';
// Wait until SELECT name, total_rows  FROM system.tables WHERE database = 'pg'; // gives 2 rows

do

-- clickhouse
select * from pg.test

Everything is fine, all 3 columns are visible id, x, y

Now at postgresql delete column

-- postgres
delete from test where id = 2;

Update will not be replicated. In logs will be

<Warning> PostgreSQLReplicaConsumer(scrapers): Table test (relation_id: 61753011) is skipped, because table schema has changed

I think, but not sure 100% it's because of this check

if (static_cast<size_t>(num_columns) + 2 != buffer.getColumnsNum()) /// +2 -- sign and version columns
{
markTableAsSkipped(relation_id, table_name);
return;
}

because no other warnings etc are seen in the log.

without GENERATED columns all is fine.

@istarkov istarkov added the potential bug To be reviewed by developers and confirmed/rejected. label Aug 20, 2022
@kssenii kssenii self-assigned this Aug 21, 2022
@kssenii kssenii added experimental feature Bug in the feature that should not be used in production and removed potential bug To be reviewed by developers and confirmed/rejected. labels Apr 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
experimental feature Bug in the feature that should not be used in production
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants