Schemer: no need to migrate schema from pod which hasn't been created

[slimtom95]

In a concile process, it's no need to migrate schema from the hosts that we know it's not exist and will be created later;

Schemer can really hang there for some time to wait invisible nodes respond.

[alex-zaitsev]

Schemer uses the load balancer and skip_unavailable_shards setting, so it should not go to inactive pods.

[Sallery-X]

the process spent too a lot time，i think when we create new cluster，no need to migrating schema objects to host

[Sallery-X]

flag that mean the old chi is nil.

[Sallery-X]

@alex-zaitsev

[slimtom95]

Please read this log screenshots, it did ask invisible pods to answer.

[alex-zaitsev]

ok. For some reason it does not fail fast in your environment and waits 10 seconds timeout every time.

[Slach]

@SalleryWang @slimtom95 , could you answer to following questions:

• according to your comments with code Schemer: no need to migrate schema from pod which hasn't been created #479 (comment)
  do you use a custom fork for clickhouse-operator?

• according to shared PDF, do you use clickhouse cluster with 32 shards with 3 replicas inside each shard?

look like your fork contains error in schemer.go and you tried get schema objects from unavailable hosts which not exists now

for example from shared PDF file

I0727 01:21:53.570763 1 schemer.go:247] Migrating schema objects to host 28-2
I0727 01:21:53.570839 1 schemer.go:193] Extracting replicated table definitions from [chi-ck-bo7s4lw7ax-31-0.ck�bo7s4lw7ax.svc.cluster.local chi-ck-bo7s4lw7ax-31-1.ck�bo7s4lw7ax.svc.cluster.local]
I0727 01:21:53.623319 1 connection.go:95] FAILED Query(http://
***:***@chi-ck-bo7s4lw7ax-0-0.ck-bo7s4lw7ax.svc.cluster.local:8123/) 
clickhouse: Code: 90, e.displayText() = DB::Exception: Empty list of 
columns passed (version 19.17.4.11 (official build))
 for SQL: SELECT DISTINCT 
database AS name, 
concat('CREATE DATABASE IF NOT EXISTS "', name, '"') AS 
create_db_query
FROM remote('chi-ck-bo7s4lw7ax-31-0.ckbo7s4lw7ax.svc.cluster.local,chi-ck-bo7s4lw7ax-31-1.ck�bo7s4lw7ax.svc.cluster.local', system, tables)
WHERE database != 'system'
SETTINGS skip_unavailable_shards = 1

I don't know why your custom operator fork tried to connect to chi-ck-bo7s4lw7ax-0-0 instead of chi-ck-bo7s4lw7ax-28-2?

[Sallery-X]

not a fork，it‘s origin

[Slach]

@SalleryWang which clickhouse-operator version did you use?

[Sallery-X]

@Slach v0.10.0

[Slach]

@SalleryWang in v0.10.0
part of code which you share in #479 (comment)

looks different than part of worker.go in current 0.10.0
https://github.com/Altinity/clickhouse-operator/blob/master/pkg/controller/chi/worker.go#L398

ok. after internal discussion we decided to improve schemer.go to avoid create tables when creating ClickHouse cluster from scratch

[Sallery-X]

fine

[Sallery-X]

I just made a little change to avoid migrating schema from pod which hasn't been created, The premise is i use cloud disk. so I think this is fine

[Sallery-X]

@Slach

[alex-zaitsev]

Fixed in 0.11.0

[alex-zaitsev]

https://github.com/Altinity/clickhouse-operator/releases/tag/0.11.0