minor: change maxStringLength default#19198
Merged
abhishekrb19 merged 3 commits intoapache:masterfrom Mar 23, 2026
Merged
Conversation
Contributor
abhishekrb19
left a comment
abhishekrb19
left a comment
There was a problem hiding this comment.
Thanks for the follow up @jaykanakiya! One comment on the validation.
| ) | ||
| { | ||
| super(name, multiValueHandling, createBitmapIndex == null ? DEFAULT_CREATE_BITMAP_INDEX : createBitmapIndex); | ||
| this.maxStringLength = maxStringLength != null && maxStringLength > 0 ? maxStringLength : getDefaultMaxStringLength(); |
Contributor
There was a problem hiding this comment.
Should this validation be updated to allow 0 values now: if (maxStringLength != null && maxStringLength < 0)
Contributor
Author
There was a problem hiding this comment.
Updated it to allow 0.
docs/configuration/index.md
Outdated
| |`druid.indexer.task.tmpStorageBytesPerTask`|Maximum number of bytes per task to be used to store temporary files on disk. This config is generally intended for internal usage. Attempts to set it are very likely to be overwritten by the TaskRunner that executes the task, so be sure of what you expect to happen before directly adjusting this configuration parameter. The config is documented here primarily to provide an understanding of what it means if/when someone sees that it has been set. A value of -1 disables this limit. |-1| | ||
| |`druid.indexer.server.maxChatRequests`|Maximum number of concurrent requests served by a task's chat handler. Set to 0 to disable limiting.|0| | ||
| |`druid.indexing.formats.maxStringLength`|Maximum number of characters to store per string dimension value. Longer values are truncated during ingestion. Does not apply to multi-value string dimensions. Set to 0 to disable. Can be overridden per-dimension using `maxStringLength` in the [dimension object](../ingestion/ingestion-spec.md#dimension-objects).|0 (no truncation)| | ||
| |`druid.indexing.formats.maxStringLength`|Maximum number of characters to store per string dimension value. Longer values are truncated during ingestion. Does not apply to multi-value string dimensions. Can be overridden per-dimension using `maxStringLength` in the [dimension object](../ingestion/ingestion-spec.md#dimension-objects).|`null` (no truncation)| |
Contributor
There was a problem hiding this comment.
Suggested change
| |`druid.indexing.formats.maxStringLength`|Maximum number of characters to store per string dimension value. Longer values are truncated during ingestion. Does not apply to multi-value string dimensions. Can be overridden per-dimension using `maxStringLength` in the [dimension object](../ingestion/ingestion-spec.md#dimension-objects).|`null` (no truncation)| | |
| |`druid.indexing.formats.maxStringLength`|Maximum number of characters to store per string dimension value. Longer values are truncated during ingestion. Does not apply to multi-value string dimensions. Can be overridden per-dimension using `maxStringLength` in the [dimension object](../ingestion/ingestion-spec.md#dimension-objects). Value must be >= 0. |`null` (no truncation)| |
docs/ingestion/ingestion-spec.md
Outdated
| | createBitmapIndex | For `string` typed dimensions, whether or not bitmap indexes should be created for the column in generated segments. Creating a bitmap index requires more storage, but speeds up certain kinds of filtering (especially equality and prefix filtering). Only supported for `string` typed dimensions. | `true` | | ||
| | multiValueHandling | For `string` typed dimensions, specifies the type of handling for [multi-value fields](../querying/multi-value-dimensions.md). Possible values are `array` (ingest string arrays as-is), `sorted_array` (sort string arrays during ingestion), and `sorted_set` (sort and de-duplicate string arrays during ingestion). This parameter is ignored for types other than `string`. | `sorted_array` | | ||
| | maxStringLength | For `string` typed dimensions, the maximum number of characters to store per value. Longer values are truncated during ingestion. Does not apply to multi-value string dimensions. Set to 0 to disable. Overrides the global [`druid.indexing.formats.maxStringLength`](../configuration/index.md#additional-peon-configuration) property. | `0` (no truncation) | | ||
| | maxStringLength | For `string` typed dimensions, the maximum number of characters to store per value. Longer values are truncated during ingestion. Does not apply to multi-value string dimensions. Overrides the global [`druid.indexing.formats.maxStringLength`](../configuration/index.md#additional-peon-configuration) property. | `null` (no truncation) | |
Contributor
There was a problem hiding this comment.
Suggested change
| | maxStringLength | For `string` typed dimensions, the maximum number of characters to store per value. Longer values are truncated during ingestion. Does not apply to multi-value string dimensions. Overrides the global [`druid.indexing.formats.maxStringLength`](../configuration/index.md#additional-peon-configuration) property. | `null` (no truncation) | | |
| | maxStringLength | For `string` typed dimensions, the maximum number of characters to store per value. Longer values are truncated during ingestion. Does not apply to multi-value string dimensions. Overrides the global [`druid.indexing.formats.maxStringLength`](../configuration/index.md#additional-peon-configuration) property. Value must be >= 0.| `null` (no truncation) | |
Contributor
Author
There was a problem hiding this comment.
Updated the docs.
processing/src/main/java/org/apache/druid/segment/StringDimensionIndexer.java
Outdated
Show resolved
Hide resolved
| { | ||
| super(name, multiValueHandling, createBitmapIndex == null ? DEFAULT_CREATE_BITMAP_INDEX : createBitmapIndex); | ||
| this.maxStringLength = maxStringLength != null && maxStringLength > 0 ? maxStringLength : getDefaultMaxStringLength(); | ||
| this.maxStringLength = maxStringLength != null ? maxStringLength : getDefaultMaxStringLength(); |
Contributor
There was a problem hiding this comment.
It will be good to call the same validation function for the per-dimension override maxStringLength as well. I think negative values will be accepted, but cause undefined behavior when truncation is applied: value.substring(0, maxStringLength)?
Consider adding a test in StringDimensionSchemaTest for this case
Contributor
Author
There was a problem hiding this comment.
Good point, added some validation and a test for this.
Contributor
There was a problem hiding this comment.
Since 0 is supported, the error message should be "non-negative integer" or "Value must be >= 0"
Contributor
Author
There was a problem hiding this comment.
Changed it to non negative integer.
| * Truncates the value to the first {@link #maxStringLength} characters if configured, otherwise returns it as-is. | ||
| */ | ||
| @Nullable | ||
| private String truncateIfNeeded(String value) |
Contributor
There was a problem hiding this comment.
Intellij is flagging this for me:
Suggested change
| private String truncateIfNeeded(String value) | |
| private String truncateIfNeeded(@Nullable String value) |
| private static Integer validateMaxStringLength(@Nullable Integer maxStringLength) | ||
| { | ||
| if (maxStringLength != null && maxStringLength < 0) { | ||
| throw new IllegalArgumentException("maxStringLength must be >= 0, got " + maxStringLength); |
Contributor
There was a problem hiding this comment.
Please use DruidException and include column name in the error message.
Contributor
Author
There was a problem hiding this comment.
Updated to use DruidException and also log column name.
abhishekrb19
approved these changes
Mar 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow up to #19146
Description
Changes default value of maxStringLength to
null.Release note
Updated the default value for druid.indexing.formats.maxStringLength to
nullfrom 0. This change also included documentation update for the same.Key changed/added classes in this PR
StringDimensionIndexerStringDimensionSchemaBuiltInTypesModule.javaThis PR has: