When tablet is loaded read tablet metadata after setting location

[keith-turner]

This is follow on work for #3350. In #3350 a new refresh column was added to indicate a hosted tablet needs to reload its metadata. The bulk import fate code (and probably compactions in the future) only set the refresh column (using a conditional mutation) if the tablet has current of future location set.

When a tablet loads it currently reads its metadata and then sets the location. This order can lead to race conditions where the bulk import code does not set a refresh column and the tablet does not read the files the bulk import code wrote. If the order is reversed and the bulk makes updates with conditional mutations that check the tablet location, then that will avoid race conditions.

For example consider the following scenarios.

1. Bulk code reads metadata for table T1 and sees it has no location
2. Tablet T1 sets its location in the metadata table
3. Bulk code attempts to add files to tablet T1 with a condition that the location is absent. This fails because the location is not absent
4. Tablet T1 reads it metadata and gets its set of files
5. The bulk code reread metadata for tablet T1 and see the location L1
6. The bulk code attempts to add files to the tablet T1 and a refresh column with a condition that the tablet location is L1
7. The bulk sends an async rpc to the tablet asking it to refresh
8. Tablet refreshes its metadata and deletes the refresh column

In the scenario above the tablet was made aware of the new bulk files. If the order of the tablet settings it location and reading its metadata above were reversed then the tablet would not have been aware of the concurrent bulk load. Also the bulk load would not have been aware of the concurrent tablet load.

Reading the metadata after setting the location ensures the following.

• Will see all changes that were made before the location was set
• Will be notified of any changes during and after the location was set The during part assumes other changes are also using conditional mutations.

When making this changes in the tablet code need to document how important the order is.