index_parallel task fails if segmentGranularity has a timeZone #9993
Comments
jihoonson
added
Area - Batch Ingestion
Bug
and removed
Uncategorized problem report
labels
|
@tarpdalton thank you for the detailed report! I don't have a concrete idea to fix the bug right now, but will take a look. |
|
@jihoonson I don't understand what's the meaning of setting |
|
@FrankChen021 it's documented here. #9894 is about |
The doc is about query granularity, although segment granularity shares the same type as query granularity, it does not explain why people need to care about timezone/origin of segment granularity. I don’t see any beneficial from these two parameters on segment granularity |
|
Yes, the doc should say it can be used for segment granularity as well. However, it is at least linked https://druid.apache.org/docs/latest/ingestion/index.html#granularityspec.
I'm not sure what you are suggesting. The timezone is useful when you have timestamps of a different timezone from the one where your druid is running. The origin is useful when you want to make time buckets differently. |
|
I'll share my use case for segment granularity. Here is my granularity spec for loading some data: "granularitySpec": {
"segmentGranularity": {
"type": "period",
"period": "P1D",
"timeZone": "America/New_York"
},
"queryGranularity": {
"type": "period",
"period": "P1D",
"timeZone": "America/New_York"
},
"rollup": true,
"intervals": [
"2020-05-12T00:00:00-04:00/2020-05-13T00:00:00-04:00"
]
},I am rolling up in daily buckets, but offset by the timezone. The granularity is big so the roll up is more efficient. |
|
@tarpdalton I see there's some benefits by setting timezone for segment granularity. Each segment starts at 00:00 EDT instead of UTC, to query data within a local day exactly falls into one segment. But if segment starts at 00:00 +0, data in local time may spread in two segments. |
|
We are also facing similar issue, We are using druid 0.20.2. Issue started coming when we moved from ec2 r5.4x instance to i3en.4x instance ` "tuningConfig": {
|
Affected Version
0.18.0 and 0.18.1
Description
Cluster size
Steps to reproduce the problem
index_paralleltasktimeZonein thesegmentGranularityin thegranularitySpecin thedataSchemamaxNumConcurrentSubTasksgreater than1in thetuningConfigtypeashashedforpartitionsSpecintuningConfigThe error message or stack traces encountered.
The main error is the
ZipExceptionThe unzip fails because
findPartitionFilefails to find the partition created during thepartial_index_generatetask.getPartitionreturns the error message instead of the zip file. So the unzip fails.The partition file is stored with the timezone offset in the path like this:
2020-04-24T00:00:00.000-04:00/2020-04-25T00:00:00.000-04:00But the http request to
getPartitionuses the UTC timestartTime=2020-04-24T04:00:00.000Z&endTime=2020-04-25T04:00:00.000ZAny debugging that you have already done
I'm not very familiar with the druid code so I'm not sure if there is a simple code fix. @jihoonson might know how to fix it, since he is working on #8061.
It looks like
startTimeandendTimeparam args are fromMaybe you could store
intervalwith the tz offset instead of the materialized UTC time?The text was updated successfully, but these errors were encountered: