Date intervals in Loki queries too wide?

[zarembat]

Our logs are taking quite a lot of time to load from Loki. I suppose it's caused by a very wide time range in which they are searched for in Loki:

start = ti.start_date - timedelta(days=15)
#if the task is running or queued, the task will not have end_date, in that
# case, we will use a resonable internal of 5 days

end_date = ti.end_date  or ti.start_date + timedelta(days=5)

end = end_date + timedelta(hours=1)

Why do we need to search for logs up to 15 days before task's start_date? Shouldn't it be just the start_date?

Also it would be nice to be able to parametrize the end_date for non-finished tasks (currently hard-coded to 5 days but that may be too much depending on the use case).

[snjypl]

@zarembat everytime the task is retried the start_date will get updated to the current time. so there is no way to know the time range for the older tries. so we query for the last 15 days.

if we retry the task after say 30 days. then it won't be possible to get the logs for the older task tries.

i agree that we can make the range offset configurable.

please feel free to open a PR with the changes or i will make the changes when i get a chance.

[ckljohn]

I found that you can change Loki setting split_queries_by_interval. The default value is 15m which means every query sends to loki splits into 15 * 24 * 4 (1440) sub queries. I changed to 24h in my environment and it helps a lot.