[DocDB] Fix tablet guardrail mechanism and turn it on for universes with 16 GB or less per-node RAM

[mdbridge]

Jira Link: DB-9663

Description

We built a guardrail mechanism to limit the number of tablets users can create -- see #16177.

However, it is currently broken and cannot be safely used because it requires setting a limit for the per tablet overhead memory tracker, which would cause issues because:

• the memory tracker hierarchy under per-tablet-overhead is incorrect, containing MemTable trackers, which are not per tablet, and failing to contain untracked memory, which is almost all per tablet overhead
• the current code for checking for memory pressure double counts soft memory limit checks at every level of the memory hierarchy that has a limit
• we should not be enforcing a soft memory limit for any but the root memory tracker as the point of the soft memory limit is to absorb server-wide spikes not control individual memory components

This task is to work around this issue by temporarily making the Google flag:

~/code/yugabyte-db/src/yb/tserver/tablet_memory_manager.cc:55:
DEFINE_NON_RUNTIME_int32(
    tablet_overhead_size_percentage, 0,
    "Percentage of total available memory to use for tablet-related overheads. Default is 0, "
    "meaning no limit. Must be between 0 and 100 inclusive.");

not set a limit on the per-tablet overhead memory tracker. This will make it possible to turn on this Google flag without running into the previous issues. Other GitHub issues will address fixing these issues so that it will be possible to use the Google flag in the future to set a limit for that memory tracker.

We still need information on how much memory is available for per-tablet overhead for the guardrails; get that instead directly from the function that calculates what the limit would have been,

~/code/yugabyte-db/src/yb/tserver/tablet_memory_manager.cc:161:
int64 ComputeTabletOverheadLimit() {
  CHECK(0 <= FLAGS_tablet_overhead_size_percentage && FLAGS_tablet_overhead_size_percentage <= 100)
      << Format(
             "Flag tablet_overhead_size_percentage must be between 0 and 100 inclusive. Current "
             "value: $0",
             FLAGS_tablet_overhead_size_percentage);
  if (0 == FLAGS_tablet_overhead_size_percentage) return -1;
  return MemTracker::GetRootTracker()->limit() * FLAGS_tablet_overhead_size_percentage / 100;
}

Once the issues have been worked around, turn on the guardrail mechanism based on memory available for per-tablet overhead:

~/code/yugabyte-db/src/yb/master/tablet_limits.cc:23:
DEFINE_RUNTIME_int32(tablet_replicas_per_gib_limit, 0,
    "The maximum number of tablets per GiB of RAM reserved by TServers for tablet overheads the "
    "cluster can support. A non-positive number means no limit.");
TAG_FLAG(tablet_replicas_per_gib_limit, experimental);

DEFINE_RUNTIME_int32(tablet_replicas_per_core_limit, 0,
    "The maximum number of tablets per vCPU available to TServer processes the cluster can "
    "support. A non-positive number means no limit.");
TAG_FLAG(tablet_replicas_per_core_limit, experimental);

Here, set the first flag to 1024/0.7 for now (this is based on experiments).

Discussion is still pending whether to enable this feature via a auto flag that turns on for new universes or simply via a preview flag for now. In either case, create a new boolean enable flag as it is advised to avoid integer auto flags; also it allows the customer to just turn the feature on without having to worry about what the default limits should be.

Backport to 2.20. Possibly the backport won't turn the feature on unlike trunk.

Other possibly useful code snippets:

~/code/yugabyte-db/src/yb/tserver/tablet_memory_manager.cc:181:
  tablets_overhead_mem_tracker_ = MemTracker::FindOrCreateTracker(
      ComputeTabletOverheadLimit(), "Tablets_overhead", server_mem_tracker_);

~/code/yugabyte-db/src/yb/tserver/heartbeater.cc:403:
    auto tracker =
        server_->tablet_manager()->tablet_memory_manager()->tablets_overhead_mem_tracker();
    // Only set the tablet overhead limit if the tablet overheads memory tracker exists and has a
    // limit set.  The flag to set the memory tracker's limit is tablet_overhead_size_percentage.
    if (tracker && tracker->has_limit()) {
      resources->set_tablet_overhead_ram_in_bytes(tracker->limit());
    }
  }

Issue Type

kind/new-feature

Warning: Please confirm that this issue does not contain any sensitive information

• I confirm this issue does not contain any sensitive information.

[mdbridge]

UPDATE: tentative decision is to turn this on via #20664 for 16 GB or less boxes