20.1 release blockers list

[jordanlewis]

PSA: The 20.1 release branch is now cut. The lucky winner SHA is 1225203.

As we entering an important stabilization period, do not backport anything into the release-20.1 branch unless it is part of the release blockers list. Please also add a comment if it blocks the beta releases.

Possible blockers for the 20.1 release.

General

• Mint the 20.1 cluster version before picking 20.1 SHA [rel-eng / @dt ]

AppDev

Bulk IO

• sql: nodelocal upload and export need to require admin user sql: nodelocal upload needs to require admin user #46222 [@dt]
• importccl,gcjob: failed and canceled imports don't seem to clean up their data importccl,gcjob: failed and canceled imports don't seem to clean up their data #46684 [@pbardea]
• telemetry counters on new features (very low risk, high impact) [Backup Telemetry: Identify whether a backup was full or incremental #46518, Backup Telemetry: Locality aware backup vs. not #46520, Backup Telemetry: Ability to measure performance of backups #46521]
• (possible, still investigating) incremental full cluster backup sometimes flakes? release blocker, not beta blocker. backupccl: Incremental backup fails with error ERROR: invalid previous backups (a new full backup may be required if a table has been created, dropped or truncated): no backup covers time [1586054134.368267700,0,1586054324.310442218,0) for range [/Table/2/1,/Table/2/2) (or backups out of order) #47050
• backupccl: 2.1 (and prior) backups of tables with columns of a certain type had corrupted column types when restored into 20.1. [sql: restoring 2.1 backup into 20.1 cluster succeeds but seems to be in a bad state #44453]

Admin UI (Observability)

• ui: statement diagnostics should allow downloading of bundle ui: statement diagnostics should allow downloading of bundle #46331
• ui: statements page columns only sort per page ui: statements page columns only sort per page #46173 (bug)
• ui: time dropdown on metrics page reverts to 6h randomly ui: time dropdown on metrics page reverts to 6h randomly #46145 (bug)
• ui: truncate job description tooltip ui: truncate job description tooltip  #46078 (visual bug)
• ui: Remove statements summary from table details page ui: Remove statements summary from table details page #47079 (broken feature; replaces prior bug: V20.1.0-BETA.3 database page incorrectly filtering out statements #46584)
• ui: statements page default sort by time column not visible ui: statements page default sort by time column not visible #46427 (bug)
• ui: refactor usage of latency-parse class name in non-latency settings ui: refactor usage of latency-parse class name in non-latency settings #46445 (poorly worded but this is a regression)
• ui: Fix over view page scrolling ui: Fix over view page scrolling #46516 (bug)
• ui: update existing storage tooltips as specified in docs ui: update existing storage tooltips as specified in docs #46726 (misleading tooltips need fixing)
• ui: fix the insecure cluster indicator ui: fix the insecure cluster indicator #46844 (regression; thanks @knz!)
• ui: (fix) Tooltip component styling and props ui: (fix) Tooltip component styling and props #46854 (regression)
• Telemetry (this is for metrics in Segment for us to measure feature usage)
  • ui: add telemetry to statements list page ui: add telemetry to statements list page #45506
  • ui: add telemetry to statement details page ui: add telemetry to statement details page #45625
  • ui: add telemetry to Jobs page ui: add telemetry to Jobs page #45628
  • ui: add telemetry for pagination clicks on databases page ui: add telemetry for pagination clicks on databases page #45629
  • ui: add telemetry to network latency matrix ui: add telemetry to network latency matrix #45627
  • ui: add time picker telemetry ui: add time picker telemetry #45632

Server / UI

• ui,server: enable pain-free test-on-own-laptop user journey ui,server: enable pain-free test-on-own-laptop user journey #46420 [knz]
• sql,server: crash when connecting to a running cluster with version not initialized sql: crash when connecting to a running cluster with version not initialized #46410 [ajwerner]

KV

• TxnCoordMeta refactor broke retry/savepoint logic kv: TxnCoordMeta refactor broke sqlalchemy retry/savepoint logic #45477
• sql: rollback to savepoint after schema change can lead to deadlock sql: rollback to savepoint after schema change can lead to deadlock #24885 [andrei]
• sql: ROLLBACK TO SAVEPOINT after SFU must either release locks or fail sql: ROLLBACK TO SAVEPOINT does not release locks #46075
• kv: follower read that hits intent may get stuck in endless loop kv: follower read that hits intent may get stuck in endless loop #46148 [nathan]
• kv: we should collapse the refresh spans when exceeding the memory budget kv: we should collapse the refresh spans when exceeding the memory budget #46095 [andrei]
• kv: caller violated contract of OnLockAcquired roachtest: cdc/tpcc-1000/rangefeed=true failed #46290 [nathan]
• kv: idempotency failure across implicit commit breaks refreshing a parallel commit request kv: idempotency failure across implicit commit breaks refreshing a parallel commit request #46341 [andrei]
• kvserver: v20.1.0-beta.3: received ... results, limit was ... kvserver: v20.1.0-beta.3: received ... results, limit was ... #46652 [tbg]
• kv: don't bump ReadTimestamp on EndTxn batch with bounded read latches kv/kvserver: don't bump ReadTimestamp on EndTxn batch with bounded read latches #46625 [nathan] [beta 5 blocker]
• kv: clearrange/checks=true fails due to existing lock cannot be acquired by different transaction assertion roachtest: clearrange/checks=true failed #46460 [nathan] [beta 5 blocker]
• kv: WriteTooOldError below write timestamp assertion roachtest: scaledata/filesystem_simulator/nodes=3 failed #43273 [irfan] [beta 5 blocker]
• kv: ranged intent resolution is not batched, regresses performance on TPC-C kv: ranged intent resolution is not batched, regresses performance on TPC-C #46752 [nathan]
• roachtest: cdc/tpcc-1000/rangefeed=true failed roachtest: cdc/tpcc-1000/rangefeed=true failed #46526 [nathan]
• kv: invalid Raft truncation decision panic in cli unittests kvserver: invalid Raft truncation decision panic in cli unittests #43605 [irfan]
• kv: READ_UNCOMMITTED scan observes WriteIntentError, hits assertion kv: READ_UNCOMMITTED scan observes WriteIntentError, hits assertion #47219 [nathan]
• "kv: avoid excessively wide range tombstones during Raft snapshot reception" deemed unsafe storage: avoid excessively wide range tombstones during Raft snapshot reception #44048 (comment) [nathan]
• kv: semi-syncronous intent resolution can deadlock in loaded cluster roachtest: cdc/tpcc-1000/rangefeed=true failed #47187 [nathan]
• kv: txn recovery false positive kv: txn recovery false positive #47337 [andrei]
• kv: ranged intent resolution does not respect key limit, causes assertion failure roachtest: tpccbench/nodes=9/cpu=4/chaos/partition failed #47471 [nathan]

SQL Schema

• gcjob: fix race in gc job test gcjob: fix race in gc job test #46792
• gcjob: ensure GC job timer fires appropriately when tables are expired gcjob: ensure GC job timer fires appropriately when tables are expired #46715
• alter primary key roachtest failure roachtest: alterpk-tpcc failed #45812 High priority bug, but not a release blocker per [yahor & jordan]
• 19.2 to 20.1 migration for schema change jobs sql: 19.2 to 20.1 migration for schema change jobs #46152 [lucy/paul] [beta 5 blocker]
• sqlmigrations: create jobs to GC leftover failed import/restore jobs from 19.2 sqlmigrations: create jobs to GC leftover failed import/restore jobs from 19.2 #46818 @lucy-zhang
• bug in reverting schema changes after restarting sql: re-adopting a reverting schema change with rolled-back mutations may not work #46018 [paul/lucy]
• sql: DROP DATABASE issue with TEMP tables sql: DROP DATABASE issue with TEMP tables #46393
• sql: recent regression in stopper quiescence time sql: recent regression in stopper quiescence time #47011 [knz]
• sql: WITH HASH hash column is not null but formula can return NULL sql: WITH HASH hash column is not null but formula can return NULL #47055Removing from blockers list per @ajwerner (feature gated by flag)
• sql: upgrade from 19.x to 20.x eliminates all computed columns sql: upgrade from 19.x to 20.x eliminates all computed columns #47263
• sql: errors during schema change rollback are getting swallowed sql: errors during schema change rollback are getting swallowed #47324
• gcjob: tables are being refreshed on every system config gossip update, not just zone config updates gcjob: tables are being refreshed on every system config gossip update, not just zone config updates #47312
• sql: TestMigrateSchemaChanges/drop-table-blocked-at-after-reversing-mutations-canceled is flaky sql: TestMigrateSchemaChanges/drop-table-blocked-at-after-reversing-mutations-canceled is flaky #47532

SQL Execution

• rowexec: memory accounting for row operators seems to be incorrect rowexec: panic in HashMemRowContainer.AddRowToBucket #45975
• sql: CREATEROLE is inconsistently implemented sql: CREATEROLE is inconsistently implemented #45857
• Temporary tables aren't always GC'd sql: implement a background task for temp object cleanup #45669
• [VEC] Possible deadlock scenario colexec: possible deadlock in ExternalHashJoiner #45602
• [VEC] Disk monitoring to avoid disk usage blowup colexec: add disk monitoring #45169
• [VEC] Incorrect results of some timestamp operations. colexec: fix mixed type comparison between Timestamp and TimestampTZ #44770
  • The issue is still present but these operations are now delegated to the row execution engine.
• [VEC] UNION ALL incorrect results. sql/logictest: TestLogic failed #44188
• [VEC] TPCH Q5 incorrect output with low work memory limit colexec: incorrect output on TPCH Q5 with very low memory limit #46106
• [VEC] nil pointer in hash router shutdown roachtest: tpchvec/disk failed #46256
• sql: rollback to savepoint broken in mixed-version (19.2/20.1) clusters sql: rollback to savepoint broken in mixed-version (19.2/20.1) clusters #46372 [andrei/knz]
• rowexec: misuse of memory accounts by aggregate builtins roachtest: tpchvec/perf failed #46664
• [VEC] vector type mismatch in certain scenarios colexec: window logictest fails with unexpected interface conversion when batchSize=1  #46646
• [VEC] move disk spilling operators behind feature flag release-20.1: colexec: rename vectorize setting 192auto to auto and make it the default #46925
• [VEC] register new package with panic catcher execerror: catch panics from sql/colcontainer package #47165
• system.namespace unreadable from SQL from 19.2 nodes in mixed-version state sql: system.namespace is unreadable from 19.2 nodes in mixed version state #47167
• running workload/schemachange causes 19.2 node in mixed version cluster to crash sql: forbid TIMETZ during mixed 19.2/20.1 node clusters #47156
• backport sql: ignore soft limits on scan nodes for distsql planning #47365 (sql: ignore soft limits on scan nodes for distsql planning)
• sql: prevent time-related precision during upgrade from 19.2 -> 20.1 #47425 (sql: prevent time-related precision during upgrade from 19.2 -> 20.1)

SQL Planning

• sql: Move RBAC into the BSL license. sql: Move RBAC into the BSL license. #46042 [richard/knz]
• fix handling of write-only columns during update sql: fix handling of write-only columns during update #42459 [andyk]
• sql telemetry fixes (issues sql: add foreign keys to features table #43765, sql: hash join not present in telemetry  #46188, sql: improve the index hint telemetry  #43794; PR opt: telemetry improvements (index hints, joins, FKs) #46749) [radu]
• bundles triggered via SQL don't show up in the UI ui: EXPLAIN ANALYZE (DEBUG) results don't show up #46330 [radu]
• hard scan limit removed incorrectly opt: hard scan limit removed incorrectly #47283 [radu]

Storage

• pebble: duo of WAL issues affecting compatibility with RocksDB and recovery from crashes vendor: bump pebble to de7fc1b547d5c7f390ae3a8162289f2dc7efab6a  #47350 [peter]
• pebble: fix bloom filter incompatibility with RocksDB: storage: engine/switch/nodes=3 roachtest observes re-appearing key #47406, bloom: fix hash() to be compatible with RocksDB pebble#629 [peter]

Performance Regressions

• microbenchmark regression investigation and sign-off release: 20.1 microbenchmark regression investigation and sign-off #46502

[thoszhang]

I'm adding #46018, which is a correctness bug in schema change rollbacks, to the list. I think it should be considered blocking for the beta since it also affects the correctness of #46152.

[andreimatei]

Added kv: idempotency failure across implicit commit breaks refreshing a parallel commit request #46341

[andreimatei]

Added sql: rollback to savepoint broken in mixed-version (19.2/20.1) clusters #46372

[ajwerner]

Added importccl,gcjob: failed and canceled imports don't seem to clean up their data #46684

[andreimatei]

Added kvserver: v20.1.0-beta.3: received ... results, limit was ... #46652

[tbg]

I am mitigating #46652 by turning it into a sentry-reported error instead of a crash (https://github.com/cockroachdb/cockroach/pull/46720/files). This may not be enough to remove it from the release blocker list, though.

[tbg]

@andreimatei looks like you didn't actually add that issue to the list? Done now, though.

[nvanbenschoten]

Added the following three blockers:

• don't bump ReadTimestamp on EndTxn batch with bounded read latches kv/kvserver: don't bump ReadTimestamp on EndTxn batch with bounded read latches #46625
• clearrange/checks=true fails due to existing lock cannot be acquired by different transaction assertion roachtest: clearrange/checks=true failed #46460
• WriteTooOldError below write timestamp assertion roachtest: scaledata/filesystem_simulator/nodes=3 failed #43273

[yuzefovich]

Added two release blockers (#46664 and #46646).

[thoszhang]

Adding #46715 (PR to fix bug where table/index GC could be delayed indefinitely)

[RaduBerinde]

Added sql telemetry fixes as a "soft" blocker.

[dhartunian]

Added Admin UI/Observability blockers.

[RaduBerinde]

@dhartunian none of the diagnostics-related ones are on there.. Most importantly #46331. Are we not planning to finish those?

[dhartunian]

Added statement diagnostics bundle download to Admin UI release blockers as per @RaduBerinde's comment above.

[thoszhang]

Adding #46792 which is a test-only change that fixes a flake.

[thoszhang]

Adding #46818.

[dhartunian]

Adding two more regressions on our end. Both have PRs already up for review.

• ui: fix the insecure cluster indicator ui: fix the insecure cluster indicator #46844 (regression; thanks @knz!)
• ui: (fix) Tooltip component styling and props ui: (fix) Tooltip component styling and props #46854 (regression)

[nvanbenschoten]

Adding #46752 to the list.

[andreimatei]

I've added "sql: recent regression in stopper quiescence time #47011 [knz]". Depending on what ends up being the problem, it might turn out to not be a release blocker, but also perhaps it will be.

[andreimatei]

I've added "[ ] kvserver/closedts: setting kv.closed_timestamp.target_duration to 0 does not disable #47010 [ajwerner]"
Not a new issue.

[ajwerner]

Added sql: WITH HASH hash column is not null but formula can return NULL #47055

[irfansharif]

Added "kv: invalid Raft truncation decision panic in cli unittests #43605", which I think can cause panics out in the wild.

[miretskiy]

Will you be able to get #43605 <#43605> fix in in a day or two?

…

On Mon, Apr 6, 2020 at 11:36 AM irfan sharif ***@***.***> wrote: Added "kv: invalid Raft truncation decision panic in cli unittests #43605 <#43605>", which I think can cause panics out in the wild. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#45599 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ANA4FVFYM7752C6NBGQTB6LRLHZIPANCNFSM4K73PXKA> .

[irfansharif]

Will you be able to get #43605 fix in in a day or two?

I think so, I'm typing it up today. I was just going to downgrade the panic.

[tbg]

Checking off: kvserver: v20.1.0-beta.3: received ... results, limit was ... #46652 [tbg]

@nvanbenschoten will give this bug another look to see if he can spot a bug but either way we don't want to block the release on it. Previous code inspection plus randomized testing was not able to reproduce it. Not satisfying but better than not releasing.

[thoszhang]

I checked off #46152 (19.2 to 20.1 migration for schema change jobs) earlier. The fix for #46818 (sqlmigrations: create GC jobs for failed import/restore jobs from 19.2) is being merged right now; the backport PR is #47144.

[miretskiy]

We are all clear on the release blockers list; starting rc1

[yuzefovich]

Added the need to backport #47165 as a release blocker (it doesn't need to go into rc1 though).

[jordanlewis]

Added: system.namespace unreadable from SQL from 19.2 nodes in mixed-version state #47167

This issue causes problems for 19.2 nodes in a mixed-version cluster. The nodes will not be able to read from crdb_internal.zones, which causes problems for debug zip, some parts of the admin ui, and a few other related areas.

[otan]

adding #47156 - not important for RC.

[dt]

#47167 is a hard blocker -- it probably doesn't block a beta, but it does block release, so that means any sha we start qualifying right now would be considered a beta, not an RC.

[dt]

cc @miretskiy re above w.r.t. #47167 and RC.1

[nvanbenschoten]

Adding #47219 - which is an old bug responsible for a new assertion failing.

[miretskiy]

@nvanbenschoten is 47219 rc1 blocker as well?

[pbardea]

Adding #44453.

[nvanbenschoten]

@nvanbenschoten is 47219 rc1 blocker as well?

I don't know exactly how that's being defined. It is a hard release blocker, so it should block the final release. But I don't think it should stop any intermediate release. Whether this means we should consider a currently qualifying release as a beta instead of an RC is up for discussion (#45599 (comment))

[nvanbenschoten]

Also, adding "kv: avoid excessively wide range tombstones during Raft snapshot reception" deemed unsafe #44048 (comment) as a hard release blocker. We need to back that change out of the release, as it's too risky to do anything more with it at this point in the release cycle and it hypothetically risks replica inconsistencies.

[knz]

Adding:

sql: upgrade from 19.x to 20.x eliminates all computed columns #47263

[RaduBerinde]

Added sql: hard scan limit removed incorrectly #47283. ETA for a fix is today

[thoszhang]

Adding #47324 as a possible blocker.

[yuzefovich]

Adding the need to backport #47365 (sql: ignore soft limits on scan nodes for distsql planning) as a release blocker (not RC blocker though).

[petermattis]

Adding the need to backport #47350. I think we could technically release without this backport as it only affects Pebble, but we'd have to recommend against using Pebble and it would set back our rollout plans for Pebble. These fixes should only affect Pebble.

[otan]

Adding #47425 for release blocker - not important for RC.

[nvanbenschoten]

Adding #47187 as a hard release blocker.

[andreimatei]

Added "kv: txn recovery false positive #47337 [andrei]". It's a bug causing transactions to be wrongfully committed, so it's quite bad I think.

[nvanbenschoten]

Adding #47471 as a soft release blocker. It's an old bug, but it's worse now that we perform significantly more ranged intent resolution. The fix is targetted and should be up today.

[thoszhang]

Adding #47312 as a soft release blocker. Edit: Forgot to mention that the fix is targeted and already merged to master; #47490 is the backport PR.

[thoszhang]

Adding #47532 as a potential "soft" release blocker.

For context, this is a regression I introduced as part of fixing #47324, and it'll take a one-line change to fix.

[petermattis]

Added #47406 which is fixed by cockroachdb/pebble#629. There is a subtle incompatibility between RocksDB and Pebble bloom filters which was causing the failures in #47406. While we've only seen problems in a test which switches back and forth between RocksDB and Pebble, I'm very anxious this could be causing other rare problems as Pebble is used to create sstables used for ingestion (e.g. during rebalancing, import, and restore) even with RocksDB is the storage engine in use.

[otan]

considering 20.1 released.