Fix cleanup_index_files in smoosh so we can enable it by default

[nickva]

cleanup_index_files is currently disabled by default:

couchdb/src/smoosh/src/smoosh_channel.erl

Lines 569 to 575 in ebbcc7e

	cleanup_index_files(DbName, _Shard) ->
	case config:get("smoosh", "cleanup_index_files", "false") of
	"true" ->
	fabric:cleanup_index_files(DbName);
	_ ->
	ok
	end.

When enabled it can misbehave, especially in cases when there are lot of concurrent view compactions. As implemented currently cleanup_index_files spawns a simple, unmonitored process on each shard compaction start

couchdb/src/smoosh/src/smoosh_channel.erl

Lines 498 to 501 in ebbcc7e

	spawn(fun() -> cleanup_index_files(DbName, Shard) end),
	Ref = erlang:monitor(process, Pid),
	Pid ! {'$gen_call', {self(), Ref}, compact},
	State#state{starting = [{Ref, {Shard, GroupId}} | State#state.starting]};

:

During cleanup it performs a cluster-wide fabric call to fetch all design docs and then does wildcard view index folder search for stale files. Since this happens at the cluster database level we end performing the exact same cleanup action for the same fabric db Q times. With the fetching all ddocs and the wildcard directory search this operation can quickly overwhelm the cluster, falling behind while creating these unbounded processes, often doing redundant work.

At the same time, it would be nice to have the ability to cleanup stale view index files so they don't keep piling up and being able to default this setting to true.

[nickva]

One idea how implement it is to add a separate cleanup channel to smoosh so instead spawning unmonitored processes there is channel which controls both capacity and concurrency of these cleanups. The database object would be clustered databases so if we're already enqueued clustered db db1 once when we try to enqueue it again we ignore (de-duplicate) the second request.

[nickva]

Another idea is in addition to the index cleanup also remove stale view purge checkpoints (_local docs). During often view updates those can end up left behind, and when that happens they can prevent further purge request processing. This manifests as warnings and error in the logs as seen in some recent issues like #4181

[nickva]

Index cleanup was improved in #4261

It's set to happen automatically in #4266 pending review

[nickva]

With #4266 index cleanup got its won smoosh channel and is enabled by default