Skip to content

Release 30.0.1-patch2

Choose a tag to compare

@nozjkoitop nozjkoitop released this 24 Mar 12:15

Improvements

Includes a way to limit on the number of tasks by type, running at the same time.
This limit controls the percentage or absolute number of task slots that can be allocated to specific task types.
When the limit is reached, tasks will wait for resources instead of blocking other tasks and failing due to timeouts.

New Property: taskLimits

Introduced new property taskLimits in the select strategy, configurable via the Overlord dynamic configuration.

TaskLimits properties:

  • maxSlotCountByType - A map where each key is a task type (String), and the corresponding value represents the absolute limit on the number of task slots that tasks of this type can occupy. The value is an Integer that is greater than or equal to 0. For example, a value of 5 means that tasks of this type can occupy up to 5 task slots in total. If both absolute and ratio limits are specified for the same task type, the effective limit will be the smaller of the absolute limit and the limit derived from the corresponding ratio. maxSlotCountByType = {"index_parallel": 3, "query_controller": 5}. In this example, parallel indexing tasks can occupy up to 3 task slots, and query controllers can occupy up to 5 task slots.
  • maxSlotRatioByType - A map where each key is a task type (String), and the corresponding value is a Double which should be in the range [0, 1], representing the ratio of task slots that tasks of this type can occupy. This ratio defines the proportion of total task slots a task type can use, calculated as ratio * totalSlots. If both absolute and ratio limits are specified for the same task type, the effective limit will be the smaller of the absolute limit and the limit derived from the corresponding ratio. maxSlotRatioByType = {"index_parallel": 0.5, "query_controller": 0.25}. In this example, parallel indexing tasks can occupy up to 50% of the total task slots, and query controllers can occupy up to 25% of the total task slots.

Example Usage

{
  "type": "default",
  "selectStrategy": {
    "type": "fillCapacity",
    "affinityConfig": {
      "affinity": {
        "datasource1": [
          "host2:port",
          "host1:port"
        ],
        "datasource2": [
          "host3:port"
        ]
      },
      "strong": false
    },
    "taskLimits": {
      "maxSlotCountByType": {},
      "maxSlotRatioByType": {
        "query_controller": 0.1
      }
    }
  },
  "autoScaler": null
}
  • This configuration ensures that query controller tasks cannot occupy more than 10% of the total available task slots.
  • Other task types remain unrestricted unless specified in taskLimits.

Note: This release includes all fixes and improvements from previous 30.0.1-patchX releases.