Another tserver shutdown issue

[ivakegg]

In this case, doing an accumulo-cluster stop, the shutdown process was stuck but for a different reason this time. The manager in this case was repeating the message that 1308 non metadata tablets were still hosted. I scanned the metadata table for the "loc" cf and found that they were all hosted by the same tserver. In that tserver's log, it was getting spammed with "told to unload tablet that was not being served ..." The solution was of course to kill that tserver manually and then the manager finally recovered from that situation and completed the shutdown.

Tracking one of the tablets in the tserver's log, I did not find any indication that it had unloaded the tablet. I see a compaction message, and then 40 minutes later the "told to unload tablet that was not being served" message. I did not find any indication of the tablet being loaded on any other tserver. So all I can figure is that something did not get flushed to the accumulo metadata table or the tserver that was hosting those tablets simply forgot about them. This is a little scary.

Accumulo 2.1.1

[ctubbsii]

Relates to #3512

[dlmarion]

I talked with @ivakegg , he said we are good to close this and they will open a new issue if needed.