delete master Tablet lead to Tablet crash

[xhh1989]

I0104 14:49:15.547715 151 shard_watcher.go:56] Stopping shard watch...
I0104 14:49:15.547810 151 shard_watcher.go:62] Shard watch stopped.
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xbcf711]
goroutine 323 [running]:
vitess.io/vitess/go/vt/topo.(*Server).GetTablet(0xc0002163c0, 0x2778420, 0xc002e276e0, 0x0, 0x0, 0x0, 0x0)
/vt/src/vitess.io/vitess/go/vt/topo/tablet.go:227 +0x51
vitess.io/vitess/go/vt/vttablet/tabletmanager.(*TabletManager).setMasterLocked(0xc0003b2750, 0x2778420, 0xc002e276e0, 0x0, 0x0, 0x0, 0x0, 0xc000e08301, 0x271e760, 0xc00028eaf0)
/vt/src/vitess.io/vitess/go/vt/vttablet/tabletmanager/rpc_replication.go:564 +0x200
vitess.io/vitess/go/vt/vttablet/tabletmanager.(*TabletManager).SetMaster(0xc0003b2750, 0x2778420, 0xc002e276e0, 0x0, 0x0, 0x0, 0x0, 0xc0013e0101, 0x0, 0x0)
/vt/src/vitess.io/vitess/go/vt/vttablet/tabletmanager/rpc_replication.go:482 +0x11d
vitess.io/vitess/go/vt/vttablet/tabletmanager.(*TabletManager).abortMasterTerm(0xc0003b2750, 0x27783a0, 0xc000227480, 0x0, 0x0, 0x0)
/vt/src/vitess.io/vitess/go/vt/vttablet/tabletmanager/shard_sync.go:228 +0x65b
vitess.io/vitess/go/vt/vttablet/tabletmanager.(*TabletManager).shardSyncLoop(0xc0003b2750, 0x27783a0, 0xc000227480, 0xc0003f27e0, 0xc0003f28a0)
/vt/src/vitess.io/vitess/go/vt/vttablet/tabletmanager/shard_sync.go:117 +0x632
created by vitess.io/vitess/go/vt/vttablet/tabletmanager.(*TabletManager).startShardSync
/vt/src/vitess.io/vitess/go/vt/vttablet/tabletmanager/shard_sync.go:247 +0x173

[deepthi]

Out of curiosity, why are you deleting master tablet from topo?

[xhh1989]

Out of curiosity, why are you deleting master tablet from topo?

yes, delete master Tablet while it is serving

[biostone]

Out of curiosity, why are you deleting master tablet from topo?

To simulate a software/hardware failure maybe?

[sougou]

Slack thread for context: https://vitess.slack.com/archives/C0PQY0PTK/p1609748160463200

I think the TL:DR; is that we should just gracefully exit if we discover that the tablet record is gone. And also extending this rule, we should exit if we detect that a different vttablet has taken over the record.