Reshard fails to update source shards of Materialize workflows

[jjpersch]

Overview of the Issue

Given a materialize workflow from a source keyspace to the same target keyspace,
resharding the keyspace updates the materialize workflow target shards, but fails to update the materialize workflow source shards.

Note that this works properly if the materialize workflow uses different source and target keyspaces.
Resharding the source keyspace updates the materialize workflow source shards.
Resharding the target keyspace updates the materialize workflow target shards.

Reproduction Steps

Can reproduce by adding materialize workflow to the local sharding example.

Start with the example customer keyspace to be sharded:

cd examples/local
source ../common/env.sh

./101_initial_cluster.sh
mysql < ../common/insert_commerce_data.sql
./201_customer_tablets.sh   
./202_move_tables.sh  
./203_switch_reads.sh      
./204_switch_writes.sh    
./205_clean_commerce.sh 
./301_customer_sharded.sh 

Add materialize workflow from customer keyspace to customer keyspace.

vtctldclient ApplySchema --sql '
create table if not exists customer_copy (
   customer_id bigint not null,
   email varbinary(128),
   primary key(customer_id)
) ENGINE=InnoDB;
'  customer

vtctldclient ApplyVSchema --vschema '
{
    "sharded": true,
    "vindexes": {
        "hash": {
            "type": "hash"
        }
    },
    "tables": {
        "customer": {
            "column_vindexes": [
                {
                    "column": "customer_id",
                    "name": "hash"
                }
            ],
            "auto_increment": {
                "column": "customer_id",
                "sequence": "customer_seq"
            }
        },
        "customer_copy": {
            "column_vindexes": [
                {
                    "column": "customer_id",
                    "name": "hash"
                }
            ]
        },
        "corder": {
            "column_vindexes": [
                {
                    "column": "customer_id",
                    "name": "hash"
                }
            ],
            "auto_increment": {
                "column": "order_id",
                "sequence": "order_seq"
            }
        }
    }
}
' customer

vtctldclient Materialize --workflow copy_customer_customer2customer --target-keyspace customer create --source-keyspace customer --table-settings '
[
    {
      "target_table": "customer_copy",
      "source_expression": "select customer_id, email from customer"
    }
]
'

Reshard the customer keyspace

./302_new_shards.sh
./303_reshard.sh
./304_switch_reads.sh 
./305_switch_writes.sh

Note the materialize workflow has updated the target shards, but not the source shards:

vtctldclient Materialize --workflow copy_customer_customer2customer --target-keyspace customer show

{
  "workflows": [
    {
      "name": "copy_customer_customer2customer",
      "source": {
        "keyspace": "customer",
        "shards": [
          "0"
        ]
      },
      "target": {
        "keyspace": "customer",
        "shards": [
          "-80",
          "80-"
        ]
      },
 ...
}

Binary Version

Reproduced on v18.0.2 and latest release-19 branch.

vtgate version Version: 19.0.1-SNAPSHOT (Git revision 06b843ca7741e3741fab1e439ef3cc8510d99e43 branch 'release-19') built on Tue Mar 12 18:13:59 EDT 2024 by jjpersch@jjpmbp2.local using go1.22.1 darwin/arm64

Operating System and Environment details

Reproduced on macos & amazon linux 2023.

Darwin jjpmbp2.local 23.4.0 Darwin Kernel Version 23.4.0: Wed Feb 21 21:44:43 PST 2024; root:xnu-10063.101.15~2/RELEASE_ARM64_T6000 arm64

Log Fragments

No response

[mattlord]

Thank you for the excellent test case @jjpersch ! I will start investigating this shortly.