Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

roachtest: jobs/mixed-versions failed #67587

Closed
cockroach-teamcity opened this issue Jul 14, 2021 · 13 comments · Fixed by #108472
Closed

roachtest: jobs/mixed-versions failed #67587

cockroach-teamcity opened this issue Jul 14, 2021 · 13 comments · Fixed by #108472
Assignees
Labels
branch-master Failures on the master branch. C-test-failure Broken test (automatically or manually discovered). db-cy-23 O-roachtest O-robot Originated from a bot. skipped-test T-disaster-recovery

Comments

@cockroach-teamcity
Copy link
Member

cockroach-teamcity commented Jul 14, 2021

roachtest.jobs/mixed-versions failed with artifacts on master @ 5a0b11e4208ad8dc2a957f3f70199c2600155ba6:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/jobs/mixed-versions/run_1
	mixed_version_jobs.go:224,versionupgrade.go:213,mixed_version_jobs.go:322,mixed_version_jobs.go:342,test_runner.go:765: Cluster info
		Node 1: 21.1
		Node 2: 21.1
		Node 3: 21.1
		Node 4: 21.1
		
		Unsuccessful job 675588114886000641 of type IMPORT, description IMPORT TABLE tpcc.public.warehouse (w_id INT8 NOT NULL PRIMARY KEY, w_name VARCHAR(10) NOT NULL, w_street_1 VARCHAR(20) NOT NULL, w_street_2 VARCHAR(20) NOT NULL, w_city VARCHAR(20) NOT NULL, w_state CHAR(2) NOT NULL, w_zip CHAR(9) NOT NULL, w_tax DECIMAL(4,4) NOT NULL, w_ytd DECIMAL(12,2) NOT NULL) CSV DATA ('workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=2&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=4&row-start=2&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=6&row-start=4&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=8&row-start=6&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=10&row-start=8&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --survival-goal=zone --interleaved=false --regions=[] --replicate-static-columns=false --seed=1 --warehouses=10 --deprecated-fk-indexes=false --families=false --fks=true: unknown flag: --survival-goal, coordinator 0
		Unsuccessful job 675588114886230017 of type IMPORT, description IMPORT TABLE tpcc.public.new_order (no_o_id INT8 NOT NULL, no_d_id INT8 NOT NULL, no_w_id INT8 NOT NULL, PRIMARY KEY (no_w_id, no_d_id, no_o_id)) CSV DATA ('workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=22500&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=45000&row-start=22500&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=67500&row-start=45000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=90000&row-start=67500&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --deprecated-fk-indexes=false --regions=[] --replicate-static-columns=false --warehouses=10 --families=false --fks=true --interleaved=false --seed=1 --survival-goal=zone: unknown flag: --regions, coordinator 0
		Unsuccessful job 675588114887114753 of type IMPORT, description IMPORT TABLE tpcc.public.history (rowid UUID NOT NULL DEFAULT gen_random_uuid(), h_c_id INT8 NOT NULL, h_c_d_id INT8 NOT NULL, h_c_w_id INT8 NOT NULL, h_d_id INT8 NOT NULL, h_w_id INT8 NOT NULL, h_date TIMESTAMP, h_amount DECIMAL(6,2), h_data VARCHAR(24), PRIMARY KEY (h_w_id, rowid)) CSV DATA ('workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --deprecated-fk-indexes=false --fks=true --interleaved=false --regions=[] --seed=1 --survival-goal=zone --warehouses=10 --families=false --replicate-static-columns=false: unknown flag: --regions, coordinator 0
		Unsuccessful job 675588114887147521 of type IMPORT, description IMPORT TABLE tpcc.public.district (d_id INT8 NOT NULL, d_w_id INT8 NOT NULL, d_name VARCHAR(10) NOT NULL, d_street_1 VARCHAR(20) NOT NULL, d_street_2 VARCHAR(20) NOT NULL, d_city VARCHAR(20) NOT NULL, d_state CHAR(2) NOT NULL, d_zip CHAR(9) NOT NULL, d_tax DECIMAL(4,4) NOT NULL, d_ytd DECIMAL(12,2) NOT NULL, d_next_o_id INT8 NOT NULL, PRIMARY KEY (d_w_id, d_id)) CSV DATA ('workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=25&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=50&row-start=25&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75&row-start=50&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=100&row-start=75&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --families=false --replicate-static-columns=false --survival-goal=zone --warehouses=10 --seed=1 --deprecated-fk-indexes=false --fks=true --interleaved=false --regions=[]: unknown flag: --survival-goal, coordinator 0
		Unsuccessful job 675588114888491009 of type IMPORT, description IMPORT TABLE tpcc.public.customer (c_id INT8 NOT NULL, c_d_id INT8 NOT NULL, c_w_id INT8 NOT NULL, c_first VARCHAR(16) NOT NULL, c_middle CHAR(2) NOT NULL, c_last VARCHAR(16) NOT NULL, c_street_1 VARCHAR(20) NOT NULL, c_street_2 VARCHAR(20) NOT NULL, c_city VARCHAR(20) NOT NULL, c_state CHAR(2) NOT NULL, c_zip CHAR(9) NOT NULL, c_phone CHAR(16) NOT NULL, c_since TIMESTAMP NOT NULL, c_credit CHAR(2) NOT NULL, c_credit_lim DECIMAL(12,2) NOT NULL, c_discount DECIMAL(4,4) NOT NULL, c_balance DECIMAL(12,2) NOT NULL, c_ytd_payment DECIMAL(12,2) NOT NULL, c_payment_cnt INT8 NOT NULL, c_delivery_cnt INT8 NOT NULL, c_data VARCHAR(500) NOT NULL, PRIMARY KEY (c_w_id, c_d_id, c_id), INDEX customer_idx (c_w_id, c_d_id, c_last, c_first)) CSV DATA ('workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --seed=1 --survival-goal=zone --warehouses=10 --fks=true --interleaved=false --regions=[] --replicate-static-columns=false --deprecated-fk-indexes=false --families=false: unknown flag: --survival-goal, coordinator 0
		Unsuccessful job 675588114889048065 of type IMPORT, description IMPORT TABLE tpcc.public.item (i_id INT8 NOT NULL, i_im_id INT8, i_name VARCHAR(24), i_price DECIMAL(5,2), i_data VARCHAR(50), PRIMARY KEY (i_id)) CSV DATA ('workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=25000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=50000&row-start=25000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=50000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=100000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --regions=[] --survival-goal=zone --warehouses=10 --deprecated-fk-indexes=false --families=false --fks=true --interleaved=false --replicate-static-columns=false --seed=1: unknown flag: --regions, coordinator 0
		Unsuccessful job 675588114889605121 of type IMPORT, description IMPORT TABLE tpcc.public.order_line (ol_o_id INT8 NOT NULL, ol_d_id INT8 NOT NULL, ol_w_id INT8 NOT NULL, ol_number INT8 NOT NULL, ol_i_id INT8 NOT NULL, ol_supply_w_id INT8, ol_delivery_d TIMESTAMP, ol_quantity INT8, ol_amount DECIMAL(6,2), ol_dist_info CHAR(24), PRIMARY KEY (ol_w_id, ol_d_id, ol_o_id DESC, ol_number)) CSV DATA ('workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --families=false --interleaved=false --replicate-static-columns=false --survival-goal=zone --warehouses=10 --deprecated-fk-indexes=false --regions=[] --seed=1 --fks=true: unknown flag: --survival-goal, coordinator 0
		Unsuccessful job 675588114891014145 of type IMPORT, description IMPORT TABLE tpcc.public."order" (o_id INT8 NOT NULL, o_d_id INT8 NOT NULL, o_w_id INT8 NOT NULL, o_c_id INT8, o_entry_d TIMESTAMP, o_carrier_id INT8, o_ol_cnt INT8, o_all_local INT8, PRIMARY KEY (o_w_id, o_d_id, o_id DESC), CONSTRAINT order_idx UNIQUE (o_w_id, o_d_id, o_c_id, o_id DESC) STORING (o_entry_d, o_carrier_id)) CSV DATA ('workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --seed=1 --families=false --interleaved=false --regions=[] --replicate-static-columns=false --survival-goal=zone --warehouses=10 --deprecated-fk-indexes=false --fks=true: unknown flag: --regions, coordinator 0
		Unsuccessful job 675588114892324865 of type IMPORT, description IMPORT TABLE tpcc.public.stock (s_i_id INT8 NOT NULL, s_w_id INT8 NOT NULL, s_quantity INT8, s_dist_01 CHAR(24), s_dist_02 CHAR(24), s_dist_03 CHAR(24), s_dist_04 CHAR(24), s_dist_05 CHAR(24), s_dist_06 CHAR(24), s_dist_07 CHAR(24), s_dist_08 CHAR(24), s_dist_09 CHAR(24), s_dist_10 CHAR(24), s_ytd INT8, s_order_cnt INT8, s_remote_cnt INT8, s_data VARCHAR(50), PRIMARY KEY (s_w_id, s_i_id)) CSV DATA ('workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=250000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=500000&row-start=250000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=750000&row-start=500000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=1000000&row-start=750000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --survival-goal=zone --warehouses=10 --interleaved=false --families=false --fks=true --regions=[] --replicate-static-columns=false --seed=1 --deprecated-fk-indexes=false: unknown flag: --survival-goal, coordinator 0
Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh jobs/mixed-versions

Same failure on other branches

/cc @cockroachdb/bulk-io

This test on roachdash | Improve this report!

Epic CRDB-7778

Jira issue: CRDB-8618

@cockroach-teamcity cockroach-teamcity added branch-master Failures on the master branch. C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. labels Jul 14, 2021
@pbardea pbardea removed the release-blocker Indicates a release-blocker. Use with branch-release-2x.x label to denote which branch is blocked. label Jul 14, 2021
@pbardea
Copy link
Contributor

pbardea commented Jul 14, 2021

Not a release blocker -- recently unskipped test.

@pbardea pbardea self-assigned this Jul 14, 2021
@cockroach-teamcity
Copy link
Member Author

roachtest.jobs/mixed-versions failed with artifacts on master @ 31cdc0e4e0c5dda4cf67b53b45c36faf6a24eb33:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/jobs/mixed-versions/run_1
	mixed_version_jobs.go:224,versionupgrade.go:213,mixed_version_jobs.go:322,mixed_version_jobs.go:342,test_runner.go:765: Cluster info
		Node 1: 21.1
		Node 2: 21.1
		Node 3: 21.1
		Node 4: 21.1
		
		Unsuccessful job 675871432081604609 of type IMPORT, description IMPORT TABLE tpcc.public.item (i_id INT8 NOT NULL, i_im_id INT8, i_name VARCHAR(24), i_price DECIMAL(5,2), i_data VARCHAR(50), PRIMARY KEY (i_id)) CSV DATA ('workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=25000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=50000&row-start=25000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=50000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=100000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --families=false --regions=[] --seed=1 --deprecated-fk-indexes=false --fks=true --interleaved=false --replicate-static-columns=false --survival-goal=zone --warehouses=10: unknown flag: --regions, coordinator 0
		Unsuccessful job 675871432081801217 of type IMPORT, description IMPORT TABLE tpcc.public."order" (o_id INT8 NOT NULL, o_d_id INT8 NOT NULL, o_w_id INT8 NOT NULL, o_c_id INT8, o_entry_d TIMESTAMP, o_carrier_id INT8, o_ol_cnt INT8, o_all_local INT8, PRIMARY KEY (o_w_id, o_d_id, o_id DESC), CONSTRAINT order_idx UNIQUE (o_w_id, o_d_id, o_c_id, o_id DESC) STORING (o_entry_d, o_carrier_id)) CSV DATA ('workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --fks=true --regions=[] --seed=1 --survival-goal=zone --warehouses=10 --deprecated-fk-indexes=false --families=false --interleaved=false --replicate-static-columns=false: unknown flag: --regions, coordinator 0
		Unsuccessful job 675871432082325505 of type IMPORT, description IMPORT TABLE tpcc.public.history (rowid UUID NOT NULL DEFAULT gen_random_uuid(), h_c_id INT8 NOT NULL, h_c_d_id INT8 NOT NULL, h_c_w_id INT8 NOT NULL, h_d_id INT8 NOT NULL, h_w_id INT8 NOT NULL, h_date TIMESTAMP, h_amount DECIMAL(6,2), h_data VARCHAR(24), PRIMARY KEY (h_w_id, rowid)) CSV DATA ('workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --seed=1 --deprecated-fk-indexes=false --families=false --fks=true --interleaved=false --replicate-static-columns=false --regions=[] --survival-goal=zone --warehouses=10: unknown flag: --regions, coordinator 0
		Unsuccessful job 675871432083013633 of type IMPORT, description IMPORT TABLE tpcc.public.new_order (no_o_id INT8 NOT NULL, no_d_id INT8 NOT NULL, no_w_id INT8 NOT NULL, PRIMARY KEY (no_w_id, no_d_id, no_o_id)) CSV DATA ('workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=22500&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=45000&row-start=22500&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=67500&row-start=45000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=90000&row-start=67500&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --survival-goal=zone --deprecated-fk-indexes=false --families=false --fks=true --regions=[] --seed=1 --warehouses=10 --interleaved=false --replicate-static-columns=false: unknown flag: --survival-goal, coordinator 0
		Unsuccessful job 675871432084193281 of type IMPORT, description IMPORT TABLE tpcc.public.stock (s_i_id INT8 NOT NULL, s_w_id INT8 NOT NULL, s_quantity INT8, s_dist_01 CHAR(24), s_dist_02 CHAR(24), s_dist_03 CHAR(24), s_dist_04 CHAR(24), s_dist_05 CHAR(24), s_dist_06 CHAR(24), s_dist_07 CHAR(24), s_dist_08 CHAR(24), s_dist_09 CHAR(24), s_dist_10 CHAR(24), s_ytd INT8, s_order_cnt INT8, s_remote_cnt INT8, s_data VARCHAR(50), PRIMARY KEY (s_w_id, s_i_id)) CSV DATA ('workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=250000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=500000&row-start=250000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=750000&row-start=500000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=1000000&row-start=750000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --replicate-static-columns=false --seed=1 --survival-goal=zone --deprecated-fk-indexes=false --families=false --fks=true --interleaved=false --regions=[] --warehouses=10: unknown flag: --survival-goal, coordinator 0
		Unsuccessful job 675871432084750337 of type IMPORT, description IMPORT TABLE tpcc.public.customer (c_id INT8 NOT NULL, c_d_id INT8 NOT NULL, c_w_id INT8 NOT NULL, c_first VARCHAR(16) NOT NULL, c_middle CHAR(2) NOT NULL, c_last VARCHAR(16) NOT NULL, c_street_1 VARCHAR(20) NOT NULL, c_street_2 VARCHAR(20) NOT NULL, c_city VARCHAR(20) NOT NULL, c_state CHAR(2) NOT NULL, c_zip CHAR(9) NOT NULL, c_phone CHAR(16) NOT NULL, c_since TIMESTAMP NOT NULL, c_credit CHAR(2) NOT NULL, c_credit_lim DECIMAL(12,2) NOT NULL, c_discount DECIMAL(4,4) NOT NULL, c_balance DECIMAL(12,2) NOT NULL, c_ytd_payment DECIMAL(12,2) NOT NULL, c_payment_cnt INT8 NOT NULL, c_delivery_cnt INT8 NOT NULL, c_data VARCHAR(500) NOT NULL, PRIMARY KEY (c_w_id, c_d_id, c_id), INDEX customer_idx (c_w_id, c_d_id, c_last, c_first)) CSV DATA ('workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --families=false --replicate-static-columns=false --survival-goal=zone --warehouses=10 --deprecated-fk-indexes=false --fks=true --interleaved=false --regions=[] --seed=1: unknown flag: --survival-goal, coordinator 0
		Unsuccessful job 675871432086061057 of type IMPORT, description IMPORT TABLE tpcc.public.warehouse (w_id INT8 NOT NULL PRIMARY KEY, w_name VARCHAR(10) NOT NULL, w_street_1 VARCHAR(20) NOT NULL, w_street_2 VARCHAR(20) NOT NULL, w_city VARCHAR(20) NOT NULL, w_state CHAR(2) NOT NULL, w_zip CHAR(9) NOT NULL, w_tax DECIMAL(4,4) NOT NULL, w_ytd DECIMAL(12,2) NOT NULL) CSV DATA ('workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=2&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=4&row-start=2&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=6&row-start=4&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=8&row-start=6&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=10&row-start=8&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --seed=1 --deprecated-fk-indexes=false --regions=[] --survival-goal=zone --warehouses=10 --families=false --fks=true --interleaved=false --replicate-static-columns=false: unknown flag: --regions, coordinator 0
		Unsuccessful job 675871432086454273 of type IMPORT, description IMPORT TABLE tpcc.public.order_line (ol_o_id INT8 NOT NULL, ol_d_id INT8 NOT NULL, ol_w_id INT8 NOT NULL, ol_number INT8 NOT NULL, ol_i_id INT8 NOT NULL, ol_supply_w_id INT8, ol_delivery_d TIMESTAMP, ol_quantity INT8, ol_amount DECIMAL(6,2), ol_dist_info CHAR(24), PRIMARY KEY (ol_w_id, ol_d_id, ol_o_id DESC, ol_number)) CSV DATA ('workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --replicate-static-columns=false --families=false --fks=true --regions=[] --seed=1 --survival-goal=zone --warehouses=10 --deprecated-fk-indexes=false --interleaved=false: unknown flag: --regions, coordinator 0
		Unsuccessful job 675871432088256513 of type IMPORT, description IMPORT TABLE tpcc.public.district (d_id INT8 NOT NULL, d_w_id INT8 NOT NULL, d_name VARCHAR(10) NOT NULL, d_street_1 VARCHAR(20) NOT NULL, d_street_2 VARCHAR(20) NOT NULL, d_city VARCHAR(20) NOT NULL, d_state CHAR(2) NOT NULL, d_zip CHAR(9) NOT NULL, d_tax DECIMAL(4,4) NOT NULL, d_ytd DECIMAL(12,2) NOT NULL, d_next_o_id INT8 NOT NULL, PRIMARY KEY (d_w_id, d_id)) CSV DATA ('workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=25&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=50&row-start=25&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75&row-start=50&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=100&row-start=75&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --warehouses=10 --interleaved=false --regions=[] --replicate-static-columns=false --seed=1 --deprecated-fk-indexes=false --families=false --fks=true --survival-goal=zone: unknown flag: --regions, coordinator 0
Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh jobs/mixed-versions

Same failure on other branches

/cc @cockroachdb/bulk-io

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.jobs/mixed-versions failed with artifacts on master @ 4e2ee35288faf691d655d51e315a14fffab431ac:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/jobs/mixed-versions/run_1
	mixed_version_jobs.go:224,versionupgrade.go:213,mixed_version_jobs.go:322,mixed_version_jobs.go:342,test_runner.go:765: Cluster info
		Node 1: 21.1
		Node 2: 21.1
		Node 3: 21.1
		Node 4: 21.1
		
		Unsuccessful job 676153918496604161 of type IMPORT, description IMPORT TABLE tpcc.public.new_order (no_o_id INT8 NOT NULL, no_d_id INT8 NOT NULL, no_w_id INT8 NOT NULL, PRIMARY KEY (no_w_id, no_d_id, no_o_id)) CSV DATA ('workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=22500&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=45000&row-start=22500&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=67500&row-start=45000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=90000&row-start=67500&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --survival-goal=zone --warehouses=10 --families=false --interleaved=false --regions=[] --replicate-static-columns=false --seed=1 --deprecated-fk-indexes=false --fks=true: unknown flag: --survival-goal, coordinator 0
		Unsuccessful job 676153918496964609 of type IMPORT, description IMPORT TABLE tpcc.public.stock (s_i_id INT8 NOT NULL, s_w_id INT8 NOT NULL, s_quantity INT8, s_dist_01 CHAR(24), s_dist_02 CHAR(24), s_dist_03 CHAR(24), s_dist_04 CHAR(24), s_dist_05 CHAR(24), s_dist_06 CHAR(24), s_dist_07 CHAR(24), s_dist_08 CHAR(24), s_dist_09 CHAR(24), s_dist_10 CHAR(24), s_ytd INT8, s_order_cnt INT8, s_remote_cnt INT8, s_data VARCHAR(50), PRIMARY KEY (s_w_id, s_i_id)) CSV DATA ('workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=250000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=500000&row-start=250000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=750000&row-start=500000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=1000000&row-start=750000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --survival-goal=zone --deprecated-fk-indexes=false --interleaved=false --seed=1 --warehouses=10 --families=false --fks=true --regions=[] --replicate-static-columns=false: unknown flag: --survival-goal, coordinator 0
		Unsuccessful job 676153918497030145 of type IMPORT, description IMPORT TABLE tpcc.public.warehouse (w_id INT8 NOT NULL PRIMARY KEY, w_name VARCHAR(10) NOT NULL, w_street_1 VARCHAR(20) NOT NULL, w_street_2 VARCHAR(20) NOT NULL, w_city VARCHAR(20) NOT NULL, w_state CHAR(2) NOT NULL, w_zip CHAR(9) NOT NULL, w_tax DECIMAL(4,4) NOT NULL, w_ytd DECIMAL(12,2) NOT NULL) CSV DATA ('workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=2&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=4&row-start=2&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=6&row-start=4&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=8&row-start=6&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=10&row-start=8&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --fks=true --interleaved=false --regions=[] --replicate-static-columns=false --survival-goal=zone --warehouses=10 --deprecated-fk-indexes=false --families=false --seed=1: unknown flag: --regions, coordinator 0
		Unsuccessful job 676153918497816577 of type IMPORT, description IMPORT TABLE tpcc.public.customer (c_id INT8 NOT NULL, c_d_id INT8 NOT NULL, c_w_id INT8 NOT NULL, c_first VARCHAR(16) NOT NULL, c_middle CHAR(2) NOT NULL, c_last VARCHAR(16) NOT NULL, c_street_1 VARCHAR(20) NOT NULL, c_street_2 VARCHAR(20) NOT NULL, c_city VARCHAR(20) NOT NULL, c_state CHAR(2) NOT NULL, c_zip CHAR(9) NOT NULL, c_phone CHAR(16) NOT NULL, c_since TIMESTAMP NOT NULL, c_credit CHAR(2) NOT NULL, c_credit_lim DECIMAL(12,2) NOT NULL, c_discount DECIMAL(4,4) NOT NULL, c_balance DECIMAL(12,2) NOT NULL, c_ytd_payment DECIMAL(12,2) NOT NULL, c_payment_cnt INT8 NOT NULL, c_delivery_cnt INT8 NOT NULL, c_data VARCHAR(500) NOT NULL, PRIMARY KEY (c_w_id, c_d_id, c_id), INDEX customer_idx (c_w_id, c_d_id, c_last, c_first)) CSV DATA ('workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --seed=1 --survival-goal=zone --deprecated-fk-indexes=false --families=false --fks=true --regions=[] --interleaved=false --replicate-static-columns=false --warehouses=10: unknown flag: --survival-goal, coordinator 0
		Unsuccessful job 676153918510268417 of type IMPORT, description IMPORT TABLE tpcc.public."order" (o_id INT8 NOT NULL, o_d_id INT8 NOT NULL, o_w_id INT8 NOT NULL, o_c_id INT8, o_entry_d TIMESTAMP, o_carrier_id INT8, o_ol_cnt INT8, o_all_local INT8, PRIMARY KEY (o_w_id, o_d_id, o_id DESC), CONSTRAINT order_idx UNIQUE (o_w_id, o_d_id, o_c_id, o_id DESC) STORING (o_entry_d, o_carrier_id)) CSV DATA ('workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --interleaved=false --replicate-static-columns=false --warehouses=10 --seed=1 --survival-goal=zone --deprecated-fk-indexes=false --families=false --fks=true --regions=[]: unknown flag: --survival-goal, coordinator 0
		Unsuccessful job 676153918511972353 of type IMPORT, description IMPORT TABLE tpcc.public.order_line (ol_o_id INT8 NOT NULL, ol_d_id INT8 NOT NULL, ol_w_id INT8 NOT NULL, ol_number INT8 NOT NULL, ol_i_id INT8 NOT NULL, ol_supply_w_id INT8, ol_delivery_d TIMESTAMP, ol_quantity INT8, ol_amount DECIMAL(6,2), ol_dist_info CHAR(24), PRIMARY KEY (ol_w_id, ol_d_id, ol_o_id DESC, ol_number)) CSV DATA ('workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --warehouses=10 --deprecated-fk-indexes=false --families=false --fks=true --interleaved=false --regions=[] --survival-goal=zone --replicate-static-columns=false --seed=1: unknown flag: --regions, coordinator 0
		Unsuccessful job 676153918515970049 of type IMPORT, description IMPORT TABLE tpcc.public.history (rowid UUID NOT NULL DEFAULT gen_random_uuid(), h_c_id INT8 NOT NULL, h_c_d_id INT8 NOT NULL, h_c_w_id INT8 NOT NULL, h_d_id INT8 NOT NULL, h_w_id INT8 NOT NULL, h_date TIMESTAMP, h_amount DECIMAL(6,2), h_data VARCHAR(24), PRIMARY KEY (h_w_id, rowid)) CSV DATA ('workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --warehouses=10 --interleaved=false --families=false --fks=true --regions=[] --replicate-static-columns=false --seed=1 --survival-goal=zone --deprecated-fk-indexes=false: unknown flag: --regions, coordinator 0
		Unsuccessful job 676153918521180161 of type IMPORT, description IMPORT TABLE tpcc.public.item (i_id INT8 NOT NULL, i_im_id INT8, i_name VARCHAR(24), i_price DECIMAL(5,2), i_data VARCHAR(50), PRIMARY KEY (i_id)) CSV DATA ('workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=25000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=50000&row-start=25000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=50000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=100000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --survival-goal=zone --deprecated-fk-indexes=false --families=false --fks=true --replicate-static-columns=false --interleaved=false --regions=[] --seed=1 --warehouses=10: unknown flag: --survival-goal, coordinator 0
		Unsuccessful job 676153918526652417 of type IMPORT, description IMPORT TABLE tpcc.public.district (d_id INT8 NOT NULL, d_w_id INT8 NOT NULL, d_name VARCHAR(10) NOT NULL, d_street_1 VARCHAR(20) NOT NULL, d_street_2 VARCHAR(20) NOT NULL, d_city VARCHAR(20) NOT NULL, d_state CHAR(2) NOT NULL, d_zip CHAR(9) NOT NULL, d_tax DECIMAL(4,4) NOT NULL, d_ytd DECIMAL(12,2) NOT NULL, d_next_o_id INT8 NOT NULL, PRIMARY KEY (d_w_id, d_id)) CSV DATA ('workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=25&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=50&row-start=25&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75&row-start=50&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=100&row-start=75&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --warehouses=10 --fks=true --interleaved=false --replicate-static-columns=false --seed=1 --survival-goal=zone --deprecated-fk-indexes=false --families=false --regions=[]: unknown flag: --survival-goal, coordinator 0
Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh jobs/mixed-versions

Same failure on other branches

/cc @cockroachdb/bulk-io

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.jobs/mixed-versions failed with artifacts on master @ efb4fd2b406cd47b505b4f1a1d5700fc79841469:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/jobs/mixed-versions/run_1
	mixed_version_jobs.go:224,versionupgrade.go:213,mixed_version_jobs.go:322,mixed_version_jobs.go:342,test_runner.go:765: Cluster info
		Node 1: 21.1
		Node 2: 21.1
		Node 3: 21.1
		Node 4: 21.1
		
		Unsuccessful job 677099687826718721 of type IMPORT, description IMPORT TABLE tpcc.public.new_order (no_o_id INT8 NOT NULL, no_d_id INT8 NOT NULL, no_w_id INT8 NOT NULL, PRIMARY KEY (no_w_id, no_d_id, no_o_id)) CSV DATA ('workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=22500&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=45000&row-start=22500&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=67500&row-start=45000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=90000&row-start=67500&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --fks=true --interleaved=false --regions=[] --seed=1 --deprecated-fk-indexes=false --families=false --replicate-static-columns=false --survival-goal=zone --warehouses=10: unknown flag: --regions, coordinator 0
		Unsuccessful job 677099687826882561 of type IMPORT, description IMPORT TABLE tpcc.public.item (i_id INT8 NOT NULL, i_im_id INT8, i_name VARCHAR(24), i_price DECIMAL(5,2), i_data VARCHAR(50), PRIMARY KEY (i_id)) CSV DATA ('workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=25000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=50000&row-start=25000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=50000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=100000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --seed=1 --fks=true --regions=[] --replicate-static-columns=false --survival-goal=zone --warehouses=10 --deprecated-fk-indexes=false --families=false --interleaved=false: unknown flag: --regions, coordinator 0
		Unsuccessful job 677099687827079169 of type IMPORT, description IMPORT TABLE tpcc.public.stock (s_i_id INT8 NOT NULL, s_w_id INT8 NOT NULL, s_quantity INT8, s_dist_01 CHAR(24), s_dist_02 CHAR(24), s_dist_03 CHAR(24), s_dist_04 CHAR(24), s_dist_05 CHAR(24), s_dist_06 CHAR(24), s_dist_07 CHAR(24), s_dist_08 CHAR(24), s_dist_09 CHAR(24), s_dist_10 CHAR(24), s_ytd INT8, s_order_cnt INT8, s_remote_cnt INT8, s_data VARCHAR(50), PRIMARY KEY (s_w_id, s_i_id)) CSV DATA ('workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=250000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=500000&row-start=250000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=750000&row-start=500000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=1000000&row-start=750000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --interleaved=false --regions=[] --survival-goal=zone --warehouses=10 --seed=1 --deprecated-fk-indexes=false --families=false --fks=true --replicate-static-columns=false: unknown flag: --regions, coordinator 0
		Unsuccessful job 677099687829274625 of type IMPORT, description IMPORT TABLE tpcc.public.district (d_id INT8 NOT NULL, d_w_id INT8 NOT NULL, d_name VARCHAR(10) NOT NULL, d_street_1 VARCHAR(20) NOT NULL, d_street_2 VARCHAR(20) NOT NULL, d_city VARCHAR(20) NOT NULL, d_state CHAR(2) NOT NULL, d_zip CHAR(9) NOT NULL, d_tax DECIMAL(4,4) NOT NULL, d_ytd DECIMAL(12,2) NOT NULL, d_next_o_id INT8 NOT NULL, PRIMARY KEY (d_w_id, d_id)) CSV DATA ('workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=25&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=50&row-start=25&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75&row-start=50&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=100&row-start=75&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --interleaved=false --replicate-static-columns=false --seed=1 --deprecated-fk-indexes=false --families=false --fks=true --regions=[] --survival-goal=zone --warehouses=10: unknown flag: --regions, coordinator 0
		Unsuccessful job 677099687829929985 of type IMPORT, description IMPORT TABLE tpcc.public.order_line (ol_o_id INT8 NOT NULL, ol_d_id INT8 NOT NULL, ol_w_id INT8 NOT NULL, ol_number INT8 NOT NULL, ol_i_id INT8 NOT NULL, ol_supply_w_id INT8, ol_delivery_d TIMESTAMP, ol_quantity INT8, ol_amount DECIMAL(6,2), ol_dist_info CHAR(24), PRIMARY KEY (ol_w_id, ol_d_id, ol_o_id DESC, ol_number)) CSV DATA ('workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --warehouses=10 --families=false --replicate-static-columns=false --regions=[] --seed=1 --survival-goal=zone --deprecated-fk-indexes=false --fks=true --interleaved=false: unknown flag: --regions, coordinator 0
		Unsuccessful job 677099687834648577 of type IMPORT, description IMPORT TABLE tpcc.public.customer (c_id INT8 NOT NULL, c_d_id INT8 NOT NULL, c_w_id INT8 NOT NULL, c_first VARCHAR(16) NOT NULL, c_middle CHAR(2) NOT NULL, c_last VARCHAR(16) NOT NULL, c_street_1 VARCHAR(20) NOT NULL, c_street_2 VARCHAR(20) NOT NULL, c_city VARCHAR(20) NOT NULL, c_state CHAR(2) NOT NULL, c_zip CHAR(9) NOT NULL, c_phone CHAR(16) NOT NULL, c_since TIMESTAMP NOT NULL, c_credit CHAR(2) NOT NULL, c_credit_lim DECIMAL(12,2) NOT NULL, c_discount DECIMAL(4,4) NOT NULL, c_balance DECIMAL(12,2) NOT NULL, c_ytd_payment DECIMAL(12,2) NOT NULL, c_payment_cnt INT8 NOT NULL, c_delivery_cnt INT8 NOT NULL, c_data VARCHAR(500) NOT NULL, PRIMARY KEY (c_w_id, c_d_id, c_id), INDEX customer_idx (c_w_id, c_d_id, c_last, c_first)) CSV DATA ('workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --fks=true --interleaved=false --replicate-static-columns=false --survival-goal=zone --deprecated-fk-indexes=false --families=false --warehouses=10 --regions=[] --seed=1: unknown flag: --survival-goal, coordinator 0
		Unsuccessful job 677099687835172865 of type IMPORT, description IMPORT TABLE tpcc.public.warehouse (w_id INT8 NOT NULL PRIMARY KEY, w_name VARCHAR(10) NOT NULL, w_street_1 VARCHAR(20) NOT NULL, w_street_2 VARCHAR(20) NOT NULL, w_city VARCHAR(20) NOT NULL, w_state CHAR(2) NOT NULL, w_zip CHAR(9) NOT NULL, w_tax DECIMAL(4,4) NOT NULL, w_ytd DECIMAL(12,2) NOT NULL) CSV DATA ('workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=2&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=4&row-start=2&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=6&row-start=4&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=8&row-start=6&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=10&row-start=8&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --survival-goal=zone --deprecated-fk-indexes=false --interleaved=false --replicate-static-columns=false --seed=1 --families=false --fks=true --regions=[] --warehouses=10: unknown flag: --survival-goal, coordinator 0
		Unsuccessful job 677099687837073409 of type IMPORT, description IMPORT TABLE tpcc.public."order" (o_id INT8 NOT NULL, o_d_id INT8 NOT NULL, o_w_id INT8 NOT NULL, o_c_id INT8, o_entry_d TIMESTAMP, o_carrier_id INT8, o_ol_cnt INT8, o_all_local INT8, PRIMARY KEY (o_w_id, o_d_id, o_id DESC), CONSTRAINT order_idx UNIQUE (o_w_id, o_d_id, o_c_id, o_id DESC) STORING (o_entry_d, o_carrier_id)) CSV DATA ('workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --interleaved=false --regions=[] --replicate-static-columns=false --warehouses=10 --fks=true --families=false --seed=1 --survival-goal=zone --deprecated-fk-indexes=false: unknown flag: --regions, coordinator 0
		Unsuccessful job 677099687842807809 of type IMPORT, description IMPORT TABLE tpcc.public.history (rowid UUID NOT NULL DEFAULT gen_random_uuid(), h_c_id INT8 NOT NULL, h_c_d_id INT8 NOT NULL, h_c_w_id INT8 NOT NULL, h_d_id INT8 NOT NULL, h_w_id INT8 NOT NULL, h_date TIMESTAMP, h_amount DECIMAL(6,2), h_data VARCHAR(24), PRIMARY KEY (h_w_id, rowid)) CSV DATA ('workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --fks=true --interleaved=false --replicate-static-columns=false --seed=1 --warehouses=10 --families=false --regions=[] --survival-goal=zone --deprecated-fk-indexes=false: unknown flag: --regions, coordinator 0
Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh jobs/mixed-versions

Same failure on other branches

/cc @cockroachdb/bulk-io

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.jobs/mixed-versions failed with artifacts on master @ fa1da4ac2872533007e4ed4c70127f85040ecb25:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/jobs/mixed-versions/run_1
	mixed_version_jobs.go:224,versionupgrade.go:213,mixed_version_jobs.go:322,mixed_version_jobs.go:342,test_runner.go:765: Cluster info
		Node 1: 21.1
		Node 2: 21.1
		Node 3: 21.1
		Node 4: 21.1
		
		Unsuccessful job 677099858210127873 of type IMPORT, description IMPORT TABLE tpcc.public.new_order (no_o_id INT8 NOT NULL, no_d_id INT8 NOT NULL, no_w_id INT8 NOT NULL, PRIMARY KEY (no_w_id, no_d_id, no_o_id)) CSV DATA ('workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=22500&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=45000&row-start=22500&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=67500&row-start=45000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=90000&row-start=67500&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --regions=[] --replicate-static-columns=false --survival-goal=zone --families=false --interleaved=false --seed=1 --warehouses=10 --deprecated-fk-indexes=false --fks=true: unknown flag: --regions, coordinator 0
		Unsuccessful job 677099858210422785 of type IMPORT, description IMPORT TABLE tpcc.public.district (d_id INT8 NOT NULL, d_w_id INT8 NOT NULL, d_name VARCHAR(10) NOT NULL, d_street_1 VARCHAR(20) NOT NULL, d_street_2 VARCHAR(20) NOT NULL, d_city VARCHAR(20) NOT NULL, d_state CHAR(2) NOT NULL, d_zip CHAR(9) NOT NULL, d_tax DECIMAL(4,4) NOT NULL, d_ytd DECIMAL(12,2) NOT NULL, d_next_o_id INT8 NOT NULL, PRIMARY KEY (d_w_id, d_id)) CSV DATA ('workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=25&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=50&row-start=25&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75&row-start=50&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=100&row-start=75&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --deprecated-fk-indexes=false --replicate-static-columns=false --seed=1 --survival-goal=zone --families=false --fks=true --interleaved=false --regions=[] --warehouses=10: unknown flag: --survival-goal, coordinator 0
		Unsuccessful job 677099858210455553 of type IMPORT, description IMPORT TABLE tpcc.public.order_line (ol_o_id INT8 NOT NULL, ol_d_id INT8 NOT NULL, ol_w_id INT8 NOT NULL, ol_number INT8 NOT NULL, ol_i_id INT8 NOT NULL, ol_supply_w_id INT8, ol_delivery_d TIMESTAMP, ol_quantity INT8, ol_amount DECIMAL(6,2), ol_dist_info CHAR(24), PRIMARY KEY (ol_w_id, ol_d_id, ol_o_id DESC, ol_number)) CSV DATA ('workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --warehouses=10 --interleaved=false --regions=[] --replicate-static-columns=false --seed=1 --survival-goal=zone --deprecated-fk-indexes=false --families=false --fks=true: unknown flag: --regions, coordinator 0
		Unsuccessful job 677099858214223873 of type IMPORT, description IMPORT TABLE tpcc.public.customer (c_id INT8 NOT NULL, c_d_id INT8 NOT NULL, c_w_id INT8 NOT NULL, c_first VARCHAR(16) NOT NULL, c_middle CHAR(2) NOT NULL, c_last VARCHAR(16) NOT NULL, c_street_1 VARCHAR(20) NOT NULL, c_street_2 VARCHAR(20) NOT NULL, c_city VARCHAR(20) NOT NULL, c_state CHAR(2) NOT NULL, c_zip CHAR(9) NOT NULL, c_phone CHAR(16) NOT NULL, c_since TIMESTAMP NOT NULL, c_credit CHAR(2) NOT NULL, c_credit_lim DECIMAL(12,2) NOT NULL, c_discount DECIMAL(4,4) NOT NULL, c_balance DECIMAL(12,2) NOT NULL, c_ytd_payment DECIMAL(12,2) NOT NULL, c_payment_cnt INT8 NOT NULL, c_delivery_cnt INT8 NOT NULL, c_data VARCHAR(500) NOT NULL, PRIMARY KEY (c_w_id, c_d_id, c_id), INDEX customer_idx (c_w_id, c_d_id, c_last, c_first)) CSV DATA ('workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --fks=true --seed=1 --survival-goal=zone --warehouses=10 --deprecated-fk-indexes=false --families=false --interleaved=false --regions=[] --replicate-static-columns=false: unknown flag: --survival-goal, coordinator 0
		Unsuccessful job 677099858216288257 of type IMPORT, description IMPORT TABLE tpcc.public.warehouse (w_id INT8 NOT NULL PRIMARY KEY, w_name VARCHAR(10) NOT NULL, w_street_1 VARCHAR(20) NOT NULL, w_street_2 VARCHAR(20) NOT NULL, w_city VARCHAR(20) NOT NULL, w_state CHAR(2) NOT NULL, w_zip CHAR(9) NOT NULL, w_tax DECIMAL(4,4) NOT NULL, w_ytd DECIMAL(12,2) NOT NULL) CSV DATA ('workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=2&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=4&row-start=2&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=6&row-start=4&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=8&row-start=6&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=10&row-start=8&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --survival-goal=zone --fks=true --regions=[] --seed=1 --deprecated-fk-indexes=false --families=false --interleaved=false --replicate-static-columns=false --warehouses=10: unknown flag: --survival-goal, coordinator 0
		Unsuccessful job 677099858218090497 of type IMPORT, description IMPORT TABLE tpcc.public.item (i_id INT8 NOT NULL, i_im_id INT8, i_name VARCHAR(24), i_price DECIMAL(5,2), i_data VARCHAR(50), PRIMARY KEY (i_id)) CSV DATA ('workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=25000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=50000&row-start=25000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=50000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=100000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --fks=true --interleaved=false --seed=1 --survival-goal=zone --deprecated-fk-indexes=false --regions=[] --replicate-static-columns=false --warehouses=10 --families=false: unknown flag: --survival-goal, coordinator 0
		Unsuccessful job 677099858222153729 of type IMPORT, description IMPORT TABLE tpcc.public.stock (s_i_id INT8 NOT NULL, s_w_id INT8 NOT NULL, s_quantity INT8, s_dist_01 CHAR(24), s_dist_02 CHAR(24), s_dist_03 CHAR(24), s_dist_04 CHAR(24), s_dist_05 CHAR(24), s_dist_06 CHAR(24), s_dist_07 CHAR(24), s_dist_08 CHAR(24), s_dist_09 CHAR(24), s_dist_10 CHAR(24), s_ytd INT8, s_order_cnt INT8, s_remote_cnt INT8, s_data VARCHAR(50), PRIMARY KEY (s_w_id, s_i_id)) CSV DATA ('workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=250000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=500000&row-start=250000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=750000&row-start=500000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=1000000&row-start=750000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --interleaved=false --replicate-static-columns=false --survival-goal=zone --deprecated-fk-indexes=false --families=false --fks=true --regions=[] --seed=1 --warehouses=10: unknown flag: --survival-goal, coordinator 0
		Unsuccessful job 677099858223071233 of type IMPORT, description IMPORT TABLE tpcc.public."order" (o_id INT8 NOT NULL, o_d_id INT8 NOT NULL, o_w_id INT8 NOT NULL, o_c_id INT8, o_entry_d TIMESTAMP, o_carrier_id INT8, o_ol_cnt INT8, o_all_local INT8, PRIMARY KEY (o_w_id, o_d_id, o_id DESC), CONSTRAINT order_idx UNIQUE (o_w_id, o_d_id, o_c_id, o_id DESC) STORING (o_entry_d, o_carrier_id)) CSV DATA ('workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --warehouses=10 --families=false --interleaved=false --replicate-static-columns=false --seed=1 --survival-goal=zone --deprecated-fk-indexes=false --fks=true --regions=[]: unknown flag: --survival-goal, coordinator 0
		Unsuccessful job 677099858227986433 of type IMPORT, description IMPORT TABLE tpcc.public.history (rowid UUID NOT NULL DEFAULT gen_random_uuid(), h_c_id INT8 NOT NULL, h_c_d_id INT8 NOT NULL, h_c_w_id INT8 NOT NULL, h_d_id INT8 NOT NULL, h_w_id INT8 NOT NULL, h_date TIMESTAMP, h_amount DECIMAL(6,2), h_data VARCHAR(24), PRIMARY KEY (h_w_id, rowid)) CSV DATA ('workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --regions=[] --seed=1 --survival-goal=zone --deprecated-fk-indexes=false --fks=true --interleaved=false --replicate-static-columns=false --warehouses=10 --families=false: unknown flag: --regions, coordinator 0
Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh jobs/mixed-versions

Same failure on other branches

/cc @cockroachdb/bulk-io

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.jobs/mixed-versions failed with artifacts on master @ 5a5b3dc446fcfc2d3e28b6775ae9bb1a63376210:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/jobs/mixed-versions/run_1
	mixed_version_jobs.go:224,versionupgrade.go:213,mixed_version_jobs.go:322,mixed_version_jobs.go:342,test_runner.go:765: Cluster info
		Node 1: 21.1
		Node 2: 21.1
		Node 3: 21.1
		Node 4: 21.1
		
		Unsuccessful job 677099930238484481 of type IMPORT, description IMPORT TABLE tpcc.public.new_order (no_o_id INT8 NOT NULL, no_d_id INT8 NOT NULL, no_w_id INT8 NOT NULL, PRIMARY KEY (no_w_id, no_d_id, no_o_id)) CSV DATA ('workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=22500&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=45000&row-start=22500&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=67500&row-start=45000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=90000&row-start=67500&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --deprecated-fk-indexes=false --families=false --regions=[] --replicate-static-columns=false --seed=1 --survival-goal=zone --fks=true --interleaved=false --warehouses=10: unknown flag: --regions, coordinator 0
		Unsuccessful job 677099930238550017 of type IMPORT, description IMPORT TABLE tpcc.public."order" (o_id INT8 NOT NULL, o_d_id INT8 NOT NULL, o_w_id INT8 NOT NULL, o_c_id INT8, o_entry_d TIMESTAMP, o_carrier_id INT8, o_ol_cnt INT8, o_all_local INT8, PRIMARY KEY (o_w_id, o_d_id, o_id DESC), CONSTRAINT order_idx UNIQUE (o_w_id, o_d_id, o_c_id, o_id DESC) STORING (o_entry_d, o_carrier_id)) CSV DATA ('workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --replicate-static-columns=false --interleaved=false --regions=[] --fks=true --seed=1 --survival-goal=zone --warehouses=10 --deprecated-fk-indexes=false --families=false: unknown flag: --regions, coordinator 0
		Unsuccessful job 677099930239172609 of type IMPORT, description IMPORT TABLE tpcc.public.customer (c_id INT8 NOT NULL, c_d_id INT8 NOT NULL, c_w_id INT8 NOT NULL, c_first VARCHAR(16) NOT NULL, c_middle CHAR(2) NOT NULL, c_last VARCHAR(16) NOT NULL, c_street_1 VARCHAR(20) NOT NULL, c_street_2 VARCHAR(20) NOT NULL, c_city VARCHAR(20) NOT NULL, c_state CHAR(2) NOT NULL, c_zip CHAR(9) NOT NULL, c_phone CHAR(16) NOT NULL, c_since TIMESTAMP NOT NULL, c_credit CHAR(2) NOT NULL, c_credit_lim DECIMAL(12,2) NOT NULL, c_discount DECIMAL(4,4) NOT NULL, c_balance DECIMAL(12,2) NOT NULL, c_ytd_payment DECIMAL(12,2) NOT NULL, c_payment_cnt INT8 NOT NULL, c_delivery_cnt INT8 NOT NULL, c_data VARCHAR(500) NOT NULL, PRIMARY KEY (c_w_id, c_d_id, c_id), INDEX customer_idx (c_w_id, c_d_id, c_last, c_first)) CSV DATA ('workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --warehouses=10 --families=false --fks=true --seed=1 --survival-goal=zone --deprecated-fk-indexes=false --interleaved=false --regions=[] --replicate-static-columns=false: unknown flag: --survival-goal, coordinator 0
		Unsuccessful job 677099930239303681 of type IMPORT, description IMPORT TABLE tpcc.public.district (d_id INT8 NOT NULL, d_w_id INT8 NOT NULL, d_name VARCHAR(10) NOT NULL, d_street_1 VARCHAR(20) NOT NULL, d_street_2 VARCHAR(20) NOT NULL, d_city VARCHAR(20) NOT NULL, d_state CHAR(2) NOT NULL, d_zip CHAR(9) NOT NULL, d_tax DECIMAL(4,4) NOT NULL, d_ytd DECIMAL(12,2) NOT NULL, d_next_o_id INT8 NOT NULL, PRIMARY KEY (d_w_id, d_id)) CSV DATA ('workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=25&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=50&row-start=25&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75&row-start=50&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=100&row-start=75&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --replicate-static-columns=false --seed=1 --families=false --interleaved=false --regions=[] --survival-goal=zone --warehouses=10 --deprecated-fk-indexes=false --fks=true: unknown flag: --regions, coordinator 0
		Unsuccessful job 677099930240286721 of type IMPORT, description IMPORT TABLE tpcc.public.stock (s_i_id INT8 NOT NULL, s_w_id INT8 NOT NULL, s_quantity INT8, s_dist_01 CHAR(24), s_dist_02 CHAR(24), s_dist_03 CHAR(24), s_dist_04 CHAR(24), s_dist_05 CHAR(24), s_dist_06 CHAR(24), s_dist_07 CHAR(24), s_dist_08 CHAR(24), s_dist_09 CHAR(24), s_dist_10 CHAR(24), s_ytd INT8, s_order_cnt INT8, s_remote_cnt INT8, s_data VARCHAR(50), PRIMARY KEY (s_w_id, s_i_id)) CSV DATA ('workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=250000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=500000&row-start=250000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=750000&row-start=500000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=1000000&row-start=750000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --survival-goal=zone --fks=true --interleaved=false --regions=[] --deprecated-fk-indexes=false --families=false --replicate-static-columns=false --seed=1 --warehouses=10: unknown flag: --survival-goal, coordinator 0
		Unsuccessful job 677099930241073153 of type IMPORT, description IMPORT TABLE tpcc.public.history (rowid UUID NOT NULL DEFAULT gen_random_uuid(), h_c_id INT8 NOT NULL, h_c_d_id INT8 NOT NULL, h_c_w_id INT8 NOT NULL, h_d_id INT8 NOT NULL, h_w_id INT8 NOT NULL, h_date TIMESTAMP, h_amount DECIMAL(6,2), h_data VARCHAR(24), PRIMARY KEY (h_w_id, rowid)) CSV DATA ('workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --deprecated-fk-indexes=false --fks=true --regions=[] --replicate-static-columns=false --survival-goal=zone --warehouses=10 --families=false --interleaved=false --seed=1: unknown flag: --regions, coordinator 0
		Unsuccessful job 677099930244907009 of type IMPORT, description IMPORT TABLE tpcc.public.warehouse (w_id INT8 NOT NULL PRIMARY KEY, w_name VARCHAR(10) NOT NULL, w_street_1 VARCHAR(20) NOT NULL, w_street_2 VARCHAR(20) NOT NULL, w_city VARCHAR(20) NOT NULL, w_state CHAR(2) NOT NULL, w_zip CHAR(9) NOT NULL, w_tax DECIMAL(4,4) NOT NULL, w_ytd DECIMAL(12,2) NOT NULL) CSV DATA ('workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=2&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=4&row-start=2&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=6&row-start=4&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=8&row-start=6&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=10&row-start=8&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --regions=[] --seed=1 --deprecated-fk-indexes=false --fks=true --interleaved=false --survival-goal=zone --warehouses=10 --families=false --replicate-static-columns=false: unknown flag: --regions, coordinator 0
		Unsuccessful job 677099930245431297 of type IMPORT, description IMPORT TABLE tpcc.public.item (i_id INT8 NOT NULL, i_im_id INT8, i_name VARCHAR(24), i_price DECIMAL(5,2), i_data VARCHAR(50), PRIMARY KEY (i_id)) CSV DATA ('workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=25000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=50000&row-start=25000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=50000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=100000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --fks=true --interleaved=false --regions=[] --seed=1 --survival-goal=zone --warehouses=10 --deprecated-fk-indexes=false --families=false --replicate-static-columns=false: unknown flag: --regions, coordinator 0
		Unsuccessful job 677099930252214273 of type IMPORT, description IMPORT TABLE tpcc.public.order_line (ol_o_id INT8 NOT NULL, ol_d_id INT8 NOT NULL, ol_w_id INT8 NOT NULL, ol_number INT8 NOT NULL, ol_i_id INT8 NOT NULL, ol_supply_w_id INT8, ol_delivery_d TIMESTAMP, ol_quantity INT8, ol_amount DECIMAL(6,2), ol_dist_info CHAR(24), PRIMARY KEY (ol_w_id, ol_d_id, ol_o_id DESC, ol_number)) CSV DATA ('workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --warehouses=10 --deprecated-fk-indexes=false --regions=[] --replicate-static-columns=false --seed=1 --survival-goal=zone --families=false --fks=true --interleaved=false: unknown flag: --regions, coordinator 0
Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh jobs/mixed-versions

Same failure on other branches

/cc @cockroachdb/bulk-io

This test on roachdash | Improve this report!

@pbardea pbardea moved this from Triage to Testing in Disaster Recovery Backlog Jul 19, 2021
@cockroach-teamcity
Copy link
Member Author

roachtest.jobs/mixed-versions failed with artifacts on master @ 5fd1dda1cde0a91643d39209c6458dcefcbfc180:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/jobs/mixed-versions/run_1
	mixed_version_jobs.go:224,versionupgrade.go:213,mixed_version_jobs.go:322,mixed_version_jobs.go:342,test_runner.go:765: Cluster info
		Node 1: 21.1
		Node 2: 21.1
		Node 3: 21.1
		Node 4: 21.1
		
		Unsuccessful job 677286174942920705 of type IMPORT, description IMPORT TABLE tpcc.public.order_line (ol_o_id INT8 NOT NULL, ol_d_id INT8 NOT NULL, ol_w_id INT8 NOT NULL, ol_number INT8 NOT NULL, ol_i_id INT8 NOT NULL, ol_supply_w_id INT8, ol_delivery_d TIMESTAMP, ol_quantity INT8, ol_amount DECIMAL(6,2), ol_dist_info CHAR(24), PRIMARY KEY (ol_w_id, ol_d_id, ol_o_id DESC, ol_number)) CSV DATA ('workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --fks=true --interleaved=false --regions=[] --seed=1 --deprecated-fk-indexes=false --families=false --replicate-static-columns=false --survival-goal=zone --warehouses=10: unknown flag: --regions, coordinator 0
		Unsuccessful job 677286174942986241 of type IMPORT, description IMPORT TABLE tpcc.public.stock (s_i_id INT8 NOT NULL, s_w_id INT8 NOT NULL, s_quantity INT8, s_dist_01 CHAR(24), s_dist_02 CHAR(24), s_dist_03 CHAR(24), s_dist_04 CHAR(24), s_dist_05 CHAR(24), s_dist_06 CHAR(24), s_dist_07 CHAR(24), s_dist_08 CHAR(24), s_dist_09 CHAR(24), s_dist_10 CHAR(24), s_ytd INT8, s_order_cnt INT8, s_remote_cnt INT8, s_data VARCHAR(50), PRIMARY KEY (s_w_id, s_i_id)) CSV DATA ('workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=250000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=500000&row-start=250000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=750000&row-start=500000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=1000000&row-start=750000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --regions=[] --seed=1 --fks=true --interleaved=false --replicate-static-columns=false --survival-goal=zone --warehouses=10 --deprecated-fk-indexes=false --families=false: unknown flag: --regions, coordinator 0
		Unsuccessful job 677286174943346689 of type IMPORT, description IMPORT TABLE tpcc.public."order" (o_id INT8 NOT NULL, o_d_id INT8 NOT NULL, o_w_id INT8 NOT NULL, o_c_id INT8, o_entry_d TIMESTAMP, o_carrier_id INT8, o_ol_cnt INT8, o_all_local INT8, PRIMARY KEY (o_w_id, o_d_id, o_id DESC), CONSTRAINT order_idx UNIQUE (o_w_id, o_d_id, o_c_id, o_id DESC) STORING (o_entry_d, o_carrier_id)) CSV DATA ('workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --fks=true --interleaved=false --regions=[] --replicate-static-columns=false --seed=1 --deprecated-fk-indexes=false --families=false --survival-goal=zone --warehouses=10: unknown flag: --regions, coordinator 0
		Unsuccessful job 677286174945968129 of type IMPORT, description IMPORT TABLE tpcc.public.district (d_id INT8 NOT NULL, d_w_id INT8 NOT NULL, d_name VARCHAR(10) NOT NULL, d_street_1 VARCHAR(20) NOT NULL, d_street_2 VARCHAR(20) NOT NULL, d_city VARCHAR(20) NOT NULL, d_state CHAR(2) NOT NULL, d_zip CHAR(9) NOT NULL, d_tax DECIMAL(4,4) NOT NULL, d_ytd DECIMAL(12,2) NOT NULL, d_next_o_id INT8 NOT NULL, PRIMARY KEY (d_w_id, d_id)) CSV DATA ('workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=25&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=50&row-start=25&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75&row-start=50&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=100&row-start=75&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --fks=true --interleaved=false --regions=[] --seed=1 --survival-goal=zone --deprecated-fk-indexes=false --families=false --replicate-static-columns=false --warehouses=10: unknown flag: --regions, coordinator 0
		Unsuccessful job 677286174946394113 of type IMPORT, description IMPORT TABLE tpcc.public.warehouse (w_id INT8 NOT NULL PRIMARY KEY, w_name VARCHAR(10) NOT NULL, w_street_1 VARCHAR(20) NOT NULL, w_street_2 VARCHAR(20) NOT NULL, w_city VARCHAR(20) NOT NULL, w_state CHAR(2) NOT NULL, w_zip CHAR(9) NOT NULL, w_tax DECIMAL(4,4) NOT NULL, w_ytd DECIMAL(12,2) NOT NULL) CSV DATA ('workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=2&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=4&row-start=2&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=6&row-start=4&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=8&row-start=6&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=10&row-start=8&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --fks=true --interleaved=false --replicate-static-columns=false --survival-goal=zone --families=false --regions=[] --seed=1 --warehouses=10 --deprecated-fk-indexes=false: unknown flag: --survival-goal, coordinator 0
		Unsuccessful job 677286174950522881 of type IMPORT, description IMPORT TABLE tpcc.public.new_order (no_o_id INT8 NOT NULL, no_d_id INT8 NOT NULL, no_w_id INT8 NOT NULL, PRIMARY KEY (no_w_id, no_d_id, no_o_id)) CSV DATA ('workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=22500&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=45000&row-start=22500&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=67500&row-start=45000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=90000&row-start=67500&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --survival-goal=zone --fks=true --interleaved=false --regions=[] --seed=1 --warehouses=10 --deprecated-fk-indexes=false --families=false --replicate-static-columns=false: unknown flag: --survival-goal, coordinator 0
		Unsuccessful job 677286174955536385 of type IMPORT, description IMPORT TABLE tpcc.public.item (i_id INT8 NOT NULL, i_im_id INT8, i_name VARCHAR(24), i_price DECIMAL(5,2), i_data VARCHAR(50), PRIMARY KEY (i_id)) CSV DATA ('workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=25000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=50000&row-start=25000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=50000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=100000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --fks=true --interleaved=false --regions=[] --replicate-static-columns=false --seed=1 --families=false --survival-goal=zone --warehouses=10 --deprecated-fk-indexes=false: unknown flag: --regions, coordinator 0
		Unsuccessful job 677286174955864065 of type IMPORT, description IMPORT TABLE tpcc.public.customer (c_id INT8 NOT NULL, c_d_id INT8 NOT NULL, c_w_id INT8 NOT NULL, c_first VARCHAR(16) NOT NULL, c_middle CHAR(2) NOT NULL, c_last VARCHAR(16) NOT NULL, c_street_1 VARCHAR(20) NOT NULL, c_street_2 VARCHAR(20) NOT NULL, c_city VARCHAR(20) NOT NULL, c_state CHAR(2) NOT NULL, c_zip CHAR(9) NOT NULL, c_phone CHAR(16) NOT NULL, c_since TIMESTAMP NOT NULL, c_credit CHAR(2) NOT NULL, c_credit_lim DECIMAL(12,2) NOT NULL, c_discount DECIMAL(4,4) NOT NULL, c_balance DECIMAL(12,2) NOT NULL, c_ytd_payment DECIMAL(12,2) NOT NULL, c_payment_cnt INT8 NOT NULL, c_delivery_cnt INT8 NOT NULL, c_data VARCHAR(500) NOT NULL, PRIMARY KEY (c_w_id, c_d_id, c_id), INDEX customer_idx (c_w_id, c_d_id, c_last, c_first)) CSV DATA ('workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --deprecated-fk-indexes=false --families=false --fks=true --interleaved=false --replicate-static-columns=false --seed=1 --warehouses=10 --regions=[] --survival-goal=zone: unknown flag: --regions, coordinator 0
		Unsuccessful job 677286174959173633 of type IMPORT, description IMPORT TABLE tpcc.public.history (rowid UUID NOT NULL DEFAULT gen_random_uuid(), h_c_id INT8 NOT NULL, h_c_d_id INT8 NOT NULL, h_c_w_id INT8 NOT NULL, h_d_id INT8 NOT NULL, h_w_id INT8 NOT NULL, h_date TIMESTAMP, h_amount DECIMAL(6,2), h_data VARCHAR(24), PRIMARY KEY (h_w_id, rowid)) CSV DATA ('workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --families=false --regions=[] --replicate-static-columns=false --survival-goal=zone --warehouses=10 --deprecated-fk-indexes=false --fks=true --interleaved=false --seed=1: unknown flag: --regions, coordinator 0
Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh jobs/mixed-versions

Same failure on other branches

/cc @cockroachdb/bulk-io

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.jobs/mixed-versions failed with artifacts on master @ 7df66cb1840c263270bd2b1a690c8e9a7c025333:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/jobs/mixed-versions/run_1
	mixed_version_jobs.go:224,versionupgrade.go:213,mixed_version_jobs.go:322,mixed_version_jobs.go:342,test_runner.go:765: Cluster info
		Node 1: 21.1
		Node 2: 21.1
		Node 3: 21.1
		Node 4: 21.1
		
		Unsuccessful job 677569366025011201 of type IMPORT, description IMPORT TABLE tpcc.public.item (i_id INT8 NOT NULL, i_im_id INT8, i_name VARCHAR(24), i_price DECIMAL(5,2), i_data VARCHAR(50), PRIMARY KEY (i_id)) CSV DATA ('workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=25000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=50000&row-start=25000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=50000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=100000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --warehouses=10 --fks=true --regions=[] --replicate-static-columns=false --survival-goal=zone --deprecated-fk-indexes=false --families=false --interleaved=false --seed=1: unknown flag: --regions, coordinator 0
		Unsuccessful job 677569366025043969 of type IMPORT, description IMPORT TABLE tpcc.public.order_line (ol_o_id INT8 NOT NULL, ol_d_id INT8 NOT NULL, ol_w_id INT8 NOT NULL, ol_number INT8 NOT NULL, ol_i_id INT8 NOT NULL, ol_supply_w_id INT8, ol_delivery_d TIMESTAMP, ol_quantity INT8, ol_amount DECIMAL(6,2), ol_dist_info CHAR(24), PRIMARY KEY (ol_w_id, ol_d_id, ol_o_id DESC, ol_number)) CSV DATA ('workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --deprecated-fk-indexes=false --families=false --interleaved=false --regions=[] --survival-goal=zone --fks=true --replicate-static-columns=false --seed=1 --warehouses=10: unknown flag: --regions, coordinator 0
		Unsuccessful job 677569366025076737 of type IMPORT, description IMPORT TABLE tpcc.public.warehouse (w_id INT8 NOT NULL PRIMARY KEY, w_name VARCHAR(10) NOT NULL, w_street_1 VARCHAR(20) NOT NULL, w_street_2 VARCHAR(20) NOT NULL, w_city VARCHAR(20) NOT NULL, w_state CHAR(2) NOT NULL, w_zip CHAR(9) NOT NULL, w_tax DECIMAL(4,4) NOT NULL, w_ytd DECIMAL(12,2) NOT NULL) CSV DATA ('workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=2&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=4&row-start=2&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=6&row-start=4&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=8&row-start=6&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=10&row-start=8&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --survival-goal=zone --deprecated-fk-indexes=false --families=false --fks=true --interleaved=false --warehouses=10 --regions=[] --replicate-static-columns=false --seed=1: unknown flag: --survival-goal, coordinator 0
		Unsuccessful job 677569366026387457 of type IMPORT, description IMPORT TABLE tpcc.public.district (d_id INT8 NOT NULL, d_w_id INT8 NOT NULL, d_name VARCHAR(10) NOT NULL, d_street_1 VARCHAR(20) NOT NULL, d_street_2 VARCHAR(20) NOT NULL, d_city VARCHAR(20) NOT NULL, d_state CHAR(2) NOT NULL, d_zip CHAR(9) NOT NULL, d_tax DECIMAL(4,4) NOT NULL, d_ytd DECIMAL(12,2) NOT NULL, d_next_o_id INT8 NOT NULL, PRIMARY KEY (d_w_id, d_id)) CSV DATA ('workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=25&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=50&row-start=25&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75&row-start=50&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=100&row-start=75&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --families=false --fks=true --survival-goal=zone --deprecated-fk-indexes=false --interleaved=false --regions=[] --replicate-static-columns=false --seed=1 --warehouses=10: unknown flag: --survival-goal, coordinator 0
		Unsuccessful job 677569366028025857 of type IMPORT, description IMPORT TABLE tpcc.public.history (rowid UUID NOT NULL DEFAULT gen_random_uuid(), h_c_id INT8 NOT NULL, h_c_d_id INT8 NOT NULL, h_c_w_id INT8 NOT NULL, h_d_id INT8 NOT NULL, h_w_id INT8 NOT NULL, h_date TIMESTAMP, h_amount DECIMAL(6,2), h_data VARCHAR(24), PRIMARY KEY (h_w_id, rowid)) CSV DATA ('workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --fks=true --survival-goal=zone --warehouses=10 --regions=[] --replicate-static-columns=false --seed=1 --deprecated-fk-indexes=false --families=false --interleaved=false: unknown flag: --survival-goal, coordinator 0
		Unsuccessful job 677569366031335425 of type IMPORT, description IMPORT TABLE tpcc.public.customer (c_id INT8 NOT NULL, c_d_id INT8 NOT NULL, c_w_id INT8 NOT NULL, c_first VARCHAR(16) NOT NULL, c_middle CHAR(2) NOT NULL, c_last VARCHAR(16) NOT NULL, c_street_1 VARCHAR(20) NOT NULL, c_street_2 VARCHAR(20) NOT NULL, c_city VARCHAR(20) NOT NULL, c_state CHAR(2) NOT NULL, c_zip CHAR(9) NOT NULL, c_phone CHAR(16) NOT NULL, c_since TIMESTAMP NOT NULL, c_credit CHAR(2) NOT NULL, c_credit_lim DECIMAL(12,2) NOT NULL, c_discount DECIMAL(4,4) NOT NULL, c_balance DECIMAL(12,2) NOT NULL, c_ytd_payment DECIMAL(12,2) NOT NULL, c_payment_cnt INT8 NOT NULL, c_delivery_cnt INT8 NOT NULL, c_data VARCHAR(500) NOT NULL, PRIMARY KEY (c_w_id, c_d_id, c_id), INDEX customer_idx (c_w_id, c_d_id, c_last, c_first)) CSV DATA ('workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --survival-goal=zone --warehouses=10 --families=false --interleaved=false --replicate-static-columns=false --deprecated-fk-indexes=false --fks=true --regions=[] --seed=1: unknown flag: --survival-goal, coordinator 0
		Unsuccessful job 677569366031368193 of type IMPORT, description IMPORT TABLE tpcc.public.stock (s_i_id INT8 NOT NULL, s_w_id INT8 NOT NULL, s_quantity INT8, s_dist_01 CHAR(24), s_dist_02 CHAR(24), s_dist_03 CHAR(24), s_dist_04 CHAR(24), s_dist_05 CHAR(24), s_dist_06 CHAR(24), s_dist_07 CHAR(24), s_dist_08 CHAR(24), s_dist_09 CHAR(24), s_dist_10 CHAR(24), s_ytd INT8, s_order_cnt INT8, s_remote_cnt INT8, s_data VARCHAR(50), PRIMARY KEY (s_w_id, s_i_id)) CSV DATA ('workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=250000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=500000&row-start=250000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=750000&row-start=500000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=1000000&row-start=750000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --regions=[] --seed=1 --deprecated-fk-indexes=false --families=false --fks=true --interleaved=false --survival-goal=zone --replicate-static-columns=false --warehouses=10: unknown flag: --regions, coordinator 0
		Unsuccessful job 677569366033104897 of type IMPORT, description IMPORT TABLE tpcc.public.new_order (no_o_id INT8 NOT NULL, no_d_id INT8 NOT NULL, no_w_id INT8 NOT NULL, PRIMARY KEY (no_w_id, no_d_id, no_o_id)) CSV DATA ('workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=22500&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=45000&row-start=22500&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=67500&row-start=45000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=90000&row-start=67500&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --deprecated-fk-indexes=false --families=false --regions=[] --survival-goal=zone --warehouses=10 --fks=true --interleaved=false --replicate-static-columns=false --seed=1: unknown flag: --regions, coordinator 0
		Unsuccessful job 677569366036774913 of type IMPORT, description IMPORT TABLE tpcc.public."order" (o_id INT8 NOT NULL, o_d_id INT8 NOT NULL, o_w_id INT8 NOT NULL, o_c_id INT8, o_entry_d TIMESTAMP, o_carrier_id INT8, o_ol_cnt INT8, o_all_local INT8, PRIMARY KEY (o_w_id, o_d_id, o_id DESC), CONSTRAINT order_idx UNIQUE (o_w_id, o_d_id, o_c_id, o_id DESC) STORING (o_entry_d, o_carrier_id)) CSV DATA ('workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --seed=1 --survival-goal=zone --warehouses=10 --fks=true --families=false --interleaved=false --regions=[] --replicate-static-columns=false --deprecated-fk-indexes=false: unknown flag: --survival-goal, coordinator 0
Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh jobs/mixed-versions

Same failure on other branches

/cc @cockroachdb/bulk-io

This test on roachdash | Improve this report!

@cockroach-teamcity
Copy link
Member Author

roachtest.jobs/mixed-versions failed with artifacts on master @ f0e2aa6abbbbf3318ea20e7dbcbe40819a809b83:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/jobs/mixed-versions/run_1
	mixed_version_jobs.go:224,versionupgrade.go:213,mixed_version_jobs.go:322,mixed_version_jobs.go:342,test_runner.go:765: Cluster info
		Node 1: 21.1
		Node 2: 21.1
		Node 3: 21.1
		Node 4: 21.1
		
		Unsuccessful job 677854245110841345 of type IMPORT, description IMPORT TABLE tpcc.public."order" (o_id INT8 NOT NULL, o_d_id INT8 NOT NULL, o_w_id INT8 NOT NULL, o_c_id INT8, o_entry_d TIMESTAMP, o_carrier_id INT8, o_ol_cnt INT8, o_all_local INT8, PRIMARY KEY (o_w_id, o_d_id, o_id DESC), CONSTRAINT order_idx UNIQUE (o_w_id, o_d_id, o_c_id, o_id DESC) STORING (o_entry_d, o_carrier_id)) CSV DATA ('workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --families=false --fks=true --seed=1 --survival-goal=zone --warehouses=10 --deprecated-fk-indexes=false --interleaved=false --regions=[] --replicate-static-columns=false: unknown flag: --survival-goal, coordinator 0
		Unsuccessful job 677854245111234561 of type IMPORT, description IMPORT TABLE tpcc.public.warehouse (w_id INT8 NOT NULL PRIMARY KEY, w_name VARCHAR(10) NOT NULL, w_street_1 VARCHAR(20) NOT NULL, w_street_2 VARCHAR(20) NOT NULL, w_city VARCHAR(20) NOT NULL, w_state CHAR(2) NOT NULL, w_zip CHAR(9) NOT NULL, w_tax DECIMAL(4,4) NOT NULL, w_ytd DECIMAL(12,2) NOT NULL) CSV DATA ('workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=2&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=4&row-start=2&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=6&row-start=4&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=8&row-start=6&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/warehouse?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=10&row-start=8&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --regions=[] --replicate-static-columns=false --deprecated-fk-indexes=false --fks=true --interleaved=false --seed=1 --survival-goal=zone --warehouses=10 --families=false: unknown flag: --regions, coordinator 0
		Unsuccessful job 677854245111300097 of type IMPORT, description IMPORT TABLE tpcc.public.customer (c_id INT8 NOT NULL, c_d_id INT8 NOT NULL, c_w_id INT8 NOT NULL, c_first VARCHAR(16) NOT NULL, c_middle CHAR(2) NOT NULL, c_last VARCHAR(16) NOT NULL, c_street_1 VARCHAR(20) NOT NULL, c_street_2 VARCHAR(20) NOT NULL, c_city VARCHAR(20) NOT NULL, c_state CHAR(2) NOT NULL, c_zip CHAR(9) NOT NULL, c_phone CHAR(16) NOT NULL, c_since TIMESTAMP NOT NULL, c_credit CHAR(2) NOT NULL, c_credit_lim DECIMAL(12,2) NOT NULL, c_discount DECIMAL(4,4) NOT NULL, c_balance DECIMAL(12,2) NOT NULL, c_ytd_payment DECIMAL(12,2) NOT NULL, c_payment_cnt INT8 NOT NULL, c_delivery_cnt INT8 NOT NULL, c_data VARCHAR(500) NOT NULL, PRIMARY KEY (c_w_id, c_d_id, c_id), INDEX customer_idx (c_w_id, c_d_id, c_last, c_first)) CSV DATA ('workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --interleaved=false --replicate-static-columns=false --survival-goal=zone --warehouses=10 --deprecated-fk-indexes=false --families=false --seed=1 --fks=true --regions=[]: unknown flag: --survival-goal, coordinator 0
		Unsuccessful job 677854245112512513 of type IMPORT, description IMPORT TABLE tpcc.public.order_line (ol_o_id INT8 NOT NULL, ol_d_id INT8 NOT NULL, ol_w_id INT8 NOT NULL, ol_number INT8 NOT NULL, ol_i_id INT8 NOT NULL, ol_supply_w_id INT8, ol_delivery_d TIMESTAMP, ol_quantity INT8, ol_amount DECIMAL(6,2), ol_dist_info CHAR(24), PRIMARY KEY (ol_w_id, ol_d_id, ol_o_id DESC, ol_number)) CSV DATA ('workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/order_line?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --regions=[] --survival-goal=zone --families=false --interleaved=false --replicate-static-columns=false --seed=1 --warehouses=10 --deprecated-fk-indexes=false --fks=true: unknown flag: --regions, coordinator 0
		Unsuccessful job 677854245112709121 of type IMPORT, description IMPORT TABLE tpcc.public.history (rowid UUID NOT NULL DEFAULT gen_random_uuid(), h_c_id INT8 NOT NULL, h_c_d_id INT8 NOT NULL, h_c_w_id INT8 NOT NULL, h_d_id INT8 NOT NULL, h_w_id INT8 NOT NULL, h_date TIMESTAMP, h_amount DECIMAL(6,2), h_data VARCHAR(24), PRIMARY KEY (h_w_id, rowid)) CSV DATA ('workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=150000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=225000&row-start=150000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/history?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=300000&row-start=225000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --warehouses=10 --deprecated-fk-indexes=false --regions=[] --seed=1 --survival-goal=zone --families=false --fks=true --interleaved=false --replicate-static-columns=false: unknown flag: --regions, coordinator 0
		Unsuccessful job 677854245119066113 of type IMPORT, description IMPORT TABLE tpcc.public.item (i_id INT8 NOT NULL, i_im_id INT8, i_name VARCHAR(24), i_price DECIMAL(5,2), i_data VARCHAR(50), PRIMARY KEY (i_id)) CSV DATA ('workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=25000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=50000&row-start=25000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75000&row-start=50000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/item?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=100000&row-start=75000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --families=false --fks=true --replicate-static-columns=false --seed=1 --deprecated-fk-indexes=false --interleaved=false --regions=[] --survival-goal=zone --warehouses=10: unknown flag: --regions, coordinator 0
		Unsuccessful job 677854245119885313 of type IMPORT, description IMPORT TABLE tpcc.public.district (d_id INT8 NOT NULL, d_w_id INT8 NOT NULL, d_name VARCHAR(10) NOT NULL, d_street_1 VARCHAR(20) NOT NULL, d_street_2 VARCHAR(20) NOT NULL, d_city VARCHAR(20) NOT NULL, d_state CHAR(2) NOT NULL, d_zip CHAR(9) NOT NULL, d_tax DECIMAL(4,4) NOT NULL, d_ytd DECIMAL(12,2) NOT NULL, d_next_o_id INT8 NOT NULL, PRIMARY KEY (d_w_id, d_id)) CSV DATA ('workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=25&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=50&row-start=25&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=75&row-start=50&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/district?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=100&row-start=75&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --regions=[] --seed=1 --survival-goal=zone --deprecated-fk-indexes=false --fks=true --replicate-static-columns=false --warehouses=10 --families=false --interleaved=false: unknown flag: --regions, coordinator 0
		Unsuccessful job 677854245120081921 of type IMPORT, description IMPORT TABLE tpcc.public.stock (s_i_id INT8 NOT NULL, s_w_id INT8 NOT NULL, s_quantity INT8, s_dist_01 CHAR(24), s_dist_02 CHAR(24), s_dist_03 CHAR(24), s_dist_04 CHAR(24), s_dist_05 CHAR(24), s_dist_06 CHAR(24), s_dist_07 CHAR(24), s_dist_08 CHAR(24), s_dist_09 CHAR(24), s_dist_10 CHAR(24), s_ytd INT8, s_order_cnt INT8, s_remote_cnt INT8, s_data VARCHAR(50), PRIMARY KEY (s_w_id, s_i_id)) CSV DATA ('workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=250000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=500000&row-start=250000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=750000&row-start=500000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/stock?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=1000000&row-start=750000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --regions=[] --warehouses=10 --seed=1 --survival-goal=zone --deprecated-fk-indexes=false --families=false --fks=true --interleaved=false --replicate-static-columns=false: unknown flag: --regions, coordinator 0
		Unsuccessful job 677854245125488641 of type IMPORT, description IMPORT TABLE tpcc.public.new_order (no_o_id INT8 NOT NULL, no_d_id INT8 NOT NULL, no_w_id INT8 NOT NULL, PRIMARY KEY (no_w_id, no_d_id, no_o_id)) CSV DATA ('workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=22500&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=45000&row-start=22500&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=67500&row-start=45000&seed=1&survival-goal=zone&version=2.2.0&warehouses=10', 'workload:///csv/tpcc/new_order?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&regions=%5B%5D&replicate-static-columns=false&row-end=90000&row-start=67500&seed=1&survival-goal=zone&version=2.2.0&warehouses=10') WITH "nullif" = 'NULL', status failed, error parsing parameters --seed=1 --survival-goal=zone --deprecated-fk-indexes=false --families=false --fks=true --regions=[] --warehouses=10 --interleaved=false --replicate-static-columns=false: unknown flag: --survival-goal, coordinator 0
Reproduce

To reproduce, try:

# From https://go.crdb.dev/p/roachstress, perhaps edited lightly.
caffeinate ./roachstress.sh jobs/mixed-versions

Same failure on other branches

/cc @cockroachdb/bulk-io

This test on roachdash | Improve this report!

@pbardea pbardea removed their assignment Jul 23, 2021
@pbardea
Copy link
Contributor

pbardea commented Jul 23, 2021

Skipped in #67890

@adityamaru adityamaru self-assigned this Jun 26, 2023
@adityamaru
Copy link
Contributor

I spoke to @renatolabs and there is value in porting this test over to the new mixed-version framework. I'll take a stab at it soon.

@shermanCRL
Copy link
Contributor

What are next steps here?

@adityamaru
Copy link
Contributor

We need to rewrite the test, I haven't gotten around to doing it yet but its on my list to close out. fwiw I think we have a lot of coverage of jobs in mixed-version tests so this is not going to add too much more to that.

craig bot pushed a commit that referenced this issue Aug 11, 2023
108472: roachtest: delete archaic jobs/mixed-version test r=dt a=adityamaru

This roachtest has dubious intentions. The expected logic on job cancellation and failure is specific to the job type. It is more useful to write job type specific unit tests rather than a roachtest to test these behaviours.

Fixes: #67587
Release note: None

108576: opt: fix index alteration recommendation for multiple invisible indexes r=rytaft a=rytaft

Fixes #108490

Release note (bug fix): Fixed a bug in the index recommendations provided in the `EXPLAIN` output where `ALTER INDEX ... VISIBLE` index recommendations may suggest making the wrong index visible when there are multiple invisible indexes in a table.

Co-authored-by: adityamaru <adityamaru@gmail.com>
Co-authored-by: Rebecca Taft <becca@cockroachlabs.com>
@craig craig bot closed this as completed in 7f60db1 Aug 11, 2023
Disaster Recovery Backlog automation moved this from Testing to Done Aug 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
branch-master Failures on the master branch. C-test-failure Broken test (automatically or manually discovered). db-cy-23 O-roachtest O-robot Originated from a bot. skipped-test T-disaster-recovery
Development

Successfully merging a pull request may close this issue.

6 participants
@pbardea @stevendanna @adityamaru @cockroach-teamcity @shermanCRL and others