Merging of accumulo.metadata tablets

[ivakegg]

We have a situation where the number of tablets exploded for the accumulo.metadata table. I expect at some point the tablet size got set too low. When trying to merge the tablets however the command reports back that you cannot do that for the accumulo.metadata table. This is very problematic. We need a way to be able to fix this system with having to manually muck with the accumulo.root table entries.

Accumulo.2.1.x
CentOD 7.3

[ctubbsii]

I just did a cursory look through the code to try to figure out why we disabled that, and couldn't find the code. I thought there was a problem that made merging the metadata table unsafe. But, now I'm not so sure, as I wasn't able to quickly find where we are blocking that. It's possible that it was only disabled because of the risks of it being offline too long for the operation, and we didn't want users shooting themselves in the foot. It might still be safe to do, with some extra precautions.

@cshannon is working on #1327 to make merges a much faster operation, where it will be much safer. I suspect it will be a big incentive to upgrade to the next LTM after 2.1, but it's too risky of a change to add to 2.1.

It might be fastest to export tables and import into a fresh Accumulo instance, but we'd have to investigate further to see whether there's another way.

[ivakegg]

@ctubbsii We have way too many tables to make exporting and importing feasible.

[ctubbsii]

We have way too many tables to make exporting and importing feasible.

Understood. Right now, I'm not sure there are any good options, though. That's the only one I know would work. Other possible solutions require further investigation.

[EdColeman]

You may be able to use the shell merge command with accumulo in SAFE mode.

accumulo org.apache.accumulo.manager.state.SetGoalState SAFE_MODE

And then in the shell the merge command with its various option can be used. Usual warnings on setting the split-sizes,...

I tried it on uno and it seemed to work as expected.

[ivakegg]

As it turns out you can actually merge the accumulo.metadata if you supply a range without being in safe mode. What does not work is using the -s (size) option. In that case you get the message "org.apache.accumulo.core.util.merge$MergeException: java.lang.IllegalArgumentException: cannot merge tablets on the metadata table."

If you can merge tablets one way, then you should be able to merge them using the size command.

[ctubbsii]

Well, that's good news! I'm glad we don't disallow merging the metadata table if needed. I was really wondering why that restriction might have been in place and couldn't think of a good one, other than to maybe prevent it being offline for awhile while the chop compactions were happening, but apparently not.

The size option triggers the use of a separate client-side utility, Merge.mergomatic() that does a lot of things to compute sizes from reading the metadata table, before incrementally issuing separate merge commands. This utility has a main method and at one point was probably used without the shell. It looks like has always had the restriction that it only works for non-metadata tables. It also predates the split of the metadata into the root and metadata tables. So, it's likely that this restriction was to prevent attempts to merge the root tablet into the rest of the metadata table. It was probably easier to impose that restriction than to account for that root tablet edge case with a bunch of complicated code. As a separate table, it's much easier to do things to the metadata table without affecting the root tablet now.

The utility would need to be updated to scan the root table instead of the metadata table when automatically computing the sizes, but otherwise, I don't see any reason the restriction couldn't be removed. I'm not entirely sure I'd recommend any kind of "auto" or "semi-auto" things happening to the metadata table, though. A utility called "mergomatic" doesn't exactly sound like the pinnacle of safety for the metadata table. 😺