[Transform] add throttling based on configuration parameter

[hendrikmuhs]

Running transform can increase load on the cluster. Especially in batch mode and for the first checkpoint of a continuous transform this can have bad side effects on the cluster.

Transform should support throttling similar to reindex. Throttling is configured in reindex with the setting requests_per_second.

Because transform works as a reducer (reindex is conceptually a mapper), transform should throttle differently:

• we be base the calculation on the number of input documents
• most runtime is spend for search/aggregations
• the number of output documents can differ, even within one transform due to data and configuration

For transform we decided to go for docs_per_second.

Setting docs_per_second to null disables throttling (instead of null you can also use -1), setting docs_per_second to 0 is disallowed.

Configuration:

{
  "source": {...},
  "dest": {...}
  "pivot": {
      "group_by": {...},
      "aggregations"" {...}
  },
  "settings": {
      "max_page_search_size": XX,
      "docs_per_second": XX
  }
}

docs_per_second is located under a new object: settings.

The setting max_page_search_size which is part of pivot will be moved there, too (the old place is deprecated but kept for BWC). If both are specified, the one under settings win, because it could be a leftover after updating an old configuration.

The whole settings object will be update-able with the update API.

Update will be

• immediate for settings
• delayed until the next checkpoint for the rest

[elasticmachine]

Pinging @elastic/ml-core (:ml/Transform)

[russcam]

Setting docs_per_second to null disables throttling

@hendrikmuhs In other APIs where behaviour can be "disabled", -1 has typically been used as a special value to signal this e.g.

• Reindex disable throttling
• Disable refresh interval on indices
• All tasks by Parent Task Id
• TimeValue MINUS_ONE for infinite timeout.

Would there be scope to disable throttling on docs_per_second by setting it to -1?

I can't speak for other language clients, but for the .NET high level client, it typically omits sending null values in a request because a null value for a property on a request type is usually one that hasn't been set by the user. We would need to write additional behaviour to be able to send an explicitly set null value in the request, to be able to distinguish it from a nullable int default value in the docs_per_second case.

Looks like the Java client sends a -1 in the case of null?

elasticsearch/client/rest-high-level/src/main/java/org/elasticsearch/client/transform/transforms/SettingsConfig.java

Lines 132 to 144 in e177a38

	/**
	* Sets the docs per second that transform can use when pulling the data from the source index.
	*
	* This setting throttles transform by issuing queries less often, however processing still happens in
	* batches. A value of 0 disables throttling (default).
	*
	* @param docsPerSecond Integer value
	* @return the {@link Builder} with requestsPerSecond set.
	*/
	public Builder setRequestsPerSecond(Float docsPerSecond) {
	this.docsPerSecond = docsPerSecond == null ? DEFAULT_DOCS_PER_SECOND : docsPerSecond;
	return this;
	}

(/cc @elastic/es-clients)

[karmi]

Agreed that null values are hard to implement (and semantically somewhat empty), and that -1, especially for numerical values, makes more sense.

[hendrikmuhs]

@russcam

Both is possible null or -1. I will update the description above.

In the stack we have both versions, null defaults are used e.g. for cluster settings.

[russcam]

Thanks for clarifying 👍 I'll document to send -1 for the client to disable