Roll-up broken on timefields not rolled up in the same time zone as Kibana

[timroes]

If you're currently trying to create a visualization on a rolled up index, while your Kibana is configured on a different timezone then the data was rolled up on (e.g. data was rolled up in UTC and you're anywhere outside UTC) the visualization will just fail.

This happens to us (in another fix) not serializing the time_zone of the AggConfig anymore when serializing to JSON so it's no longer stored in the saved object (see #34795). This unfortunately broke roll-up support, because the JSON serialization also happens to copy over the AggConfigs into some other place that actually does the query, so we're losing the time_zone already for the query.

We need to properly separate volatile parameters from "not serialized" parameters (which also effect the query).

We're still investigating if that would appear in 6.7.2, so we would mark this as a blocker, too.

[elasticmachine]

Pinging @elastic/kibana-app

[elasticmachine]

Pinging @elastic/kibana-app-arch

[lukeelmers]

Looks like this issue is present on the 6.7.2 branch, as well as master. Error message in the UI:

Rollup search error: [illegal_argument_exception] There is not a rollup job that has a [date_histogram] agg on field [@timestamp] which also satisfies all requirements of query.

To reproduce:

• Use a fresh kibana install with kibana_sample_data_logs
• Configure a rollup job on kibana_sample_data_logs, keeping the default utc timestamp
• Create index pattern for the new rollup index
• Try creating a visualization with, for example, a date histogram on the rolled up timestamp
• Error appears in the UI
• Now go to Advanced Settings and change your timezone preference to UTC
• Return to visualization, everything works as expected