backup: incorrect row counts reported by table

[dt]

Single local node, built from recent-ish master but I think I saw same thing on 21.1.

> CREATE TABLE customer (
      c_id INT8 NOT NULL,
      c_d_id INT8 NOT NULL,
      c_w_id INT8 NOT NULL,
      c_first VARCHAR(16) NOT NULL,
      c_middle CHAR(2) NOT NULL,
      c_last VARCHAR(16) NOT NULL,
      c_street_1 VARCHAR(20) NOT NULL,
      c_street_2 VARCHAR(20) NOT NULL,
      c_city VARCHAR(20) NOT NULL,
      c_state CHAR(2) NOT NULL,
      c_zip CHAR(9) NOT NULL,
      c_phone CHAR(16) NOT NULL,
      c_since TIMESTAMP NOT NULL,
      c_credit CHAR(2) NOT NULL,
      c_credit_lim DECIMAL(12,2) NOT NULL,
      c_discount DECIMAL(4,4) NOT NULL,
      c_balance DECIMAL(12,2) NOT NULL,
      c_ytd_payment DECIMAL(12,2) NOT NULL,
      c_payment_cnt INT8 NOT NULL,
      c_delivery_cnt INT8 NOT NULL,
      c_data VARCHAR(500) NOT NULL,
      CONSTRAINT "primary" PRIMARY KEY (c_w_id ASC, c_d_id ASC, c_id ASC),
      CONSTRAINT fk_c_w_id_ref_district FOREIGN KEY (c_w_id, c_d_id) REFERENCES public.district(d_w_id, d_id) NOT VALID,
      INDEX customer_idx (c_w_id ASC, c_d_id ASC, c_last ASC, c_first ASC),
      FAMILY "primary" (c_id, c_d_id, c_w_id, c_first, c_middle, c_last, c_street_1, c_street_2, c_city, c_state, c_zip, c_phone, c_since, c_credit, c_credit_lim, c_discount, c_balance, c_ytd_payment, c_payment_cnt, c_delivery_cnt, c_data)
  );


> IMPORT INTO customer CSV DATA ('workload:///csv/tpcc/customer?deprecated-fk-indexes=false&families=false&fks=true&interleaved=false&local-warehouses=false&regions=%5B%5D&replicate-static-columns=false&row-end=3000000&row-start=0&seed=1&survival-goal=zone&version=2.2.0&warehouses=100') WITH "nullif" = 'NULL'
;
        job_id       |  status   | fraction_completed |  rows   | index_entries |   bytes
---------------------+-----------+--------------------+---------+---------------+-------------
  704007449618710529 | succeeded |                  1 | 3000000 |       3000000 | 1841419174
(1 row)
...

> select count(*) from tpcc.customer;
   count
-----------
  3000000
(1 row)

> backup customer to 'nodelocal://1/c';
        job_id       |  status   | fraction_completed |  rows   | index_entries |   bytes
---------------------+-----------+--------------------+---------+---------------+-------------
  704007670454386689 | succeeded |                  1 | 3000078 |       3000005 | 1841419174
(1 row)

The row count from the IMPORT looks right, and matches what we see when we count the rows, but the row count from the backup is wrong, reporting 3000078, or 78 extra rows. I first observed this on a 8k warehouse roachprod cluster where it was off by much more, but when I tested on smaller tables, like 10k row item table, it is correct, so seems to require tables be at least some size to become incorrect?

Jira issue: CRDB-10810

gz#13539

[blathers-crl]

cc @cockroachdb/bulk-io

[dt]

It isn't off by 1 per range or anything simple like that either:

select count(*) from [show ranges from table tpcc.customer];
  count
---------
     28

[dt]

Looks like it isn't just like, first or last file either:

select path, sum(rows) from [show backup files 'nodelocal://1/tpcc.c2'] group by path ;
             path             |   sum
------------------------------+----------
  data/704009304900730881.sst | 1378405
  data/704009312552058881.sst | 1401426
  data/704009320156037121.sst |  220247

vs:

for i in data/704009312552058881.sst data/704009304900730881.sst data/704009320156037121.sst; 
  do echo $i; ./cockroach debug pebble sstable properties ./cockroach-data/extern/tpcc.c2/$i | grep records; 
done
data/704009304900730881.sst
records           1378369
data/704009312552058881.sst
records           1401389
data/704009320156037121.sst
records           3220242

That last one also includes the /2 index, so I expect it to be way off, but the first two are the pk, so they should match exactly since this is a single col family table

[msbutler]

oof. different issue.