Implement Materialized View GRACE PERIOD #15842

findepi · 2023-01-25T15:05:42Z

Description

Building on top of syntax added in #15104, fulfilling the important part of #15326, this PR delivers GRACE PERIOD support in the engine, exposing it to connectors and supporting it in Iceberg materialized views.

Additional context and related issues

Fixes #15326
Fixes #11698

Release notes

# General
* Support `GRACE PERIOD` clause in `CREATE MATERIALIZED VIEW` task.
  For backward compatibility, the existing materialized views are interpreted as
  having GRACE PERIOD of zero duration, however new materialized views have
  unlimited grace period by default. This is a backwards incompatible change and
  the previous behavior can temporarily be restored using
  `legacy.materialized-view-grace-period` configuration property or using
  `legacy_materialized_view_grace_period` session property.

# Iceberg
* Support materialized views with grace period defined.

findepi · 2023-01-25T15:19:59Z

Some preparatory commits extracted to #15843, #15844

findepi · 2023-01-27T15:12:33Z

CI #12535

findepi · 2023-03-09T15:50:42Z

just rebased, still ready for review

core/trino-main/src/main/java/io/trino/execution/CreateMaterializedViewTask.java

core/trino-main/src/main/java/io/trino/sql/analyzer/StatementAnalyzer.java

core/trino-spi/src/main/java/io/trino/spi/connector/ConnectorMaterializedViewDefinition.java

testing/trino-testing/src/main/java/io/trino/testing/BaseConnectorTest.java

core/trino-spi/pom.xml

core/trino-spi/src/main/java/io/trino/spi/connector/InMemoryRecordSet.java

plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/IcebergMetadata.java

core/trino-main/src/main/java/io/trino/sql/analyzer/StatementAnalyzer.java

losipiuk · 2023-03-10T12:40:48Z

core/trino-main/src/main/java/io/trino/sql/analyzer/StatementAnalyzer.java

+            }
+
+            // Can be negative
+            // TODO should we compare lastFreshTime with session.start() or with current time? The freshness is calculated with respect to current state of things.


Most predictable would be to cache freshness within a session and compare agains session start time.
But IDK if there are case when we compute the value within single sessions more than once. If not then it probably does not matter much.

i will leave this as is

findepi · 2023-03-10T12:47:06Z

thanks @losipiuk for you review.
Changes squashed directly since they are what you wanted them to be.
The only new code is Add Preconditions in io.trino.spi.connector package.
THere is one fixup which i mis-titled, so will force push once more.

Expose the GRACE PERIOD in `ConnectorMaterializedViewDefinition` so that connectors can store the information.

Let connector inform the engine how fresh the materialized view is. This is required to implement materialized view handling as per the new GRACE PERIOD syntax definition. Note: the property is called "last fresh time" and not "last refresh time". This is in support for connectors which know when view became no longer fresh. For example, a materialized view may have been refreshed a day ago, but only an hour ago one of its base tables changed. Then the materialized view is stale and "last fresh time" is one hour ago, not one day ago.

findepi · 2023-03-11T21:29:27Z

Error:  io.trino.plugin.sqlserver.TestSqlServerWithSnapshotIsolation.init  Time elapsed: 0.423 s  <<< FAILURE!
...
Caused by: org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception
	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:88)
	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:334)
	... 28 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container
	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:542)
	at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:344)
	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
	... 29 more
Caused by: java.lang.IllegalStateException: Container exited with code 1
	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:514)
	... 31 more

unrelated, ignored

cla-bot bot added the cla-signed label Jan 25, 2023

findepi mentioned this pull request Jan 25, 2023

Document MV GRACE PERIOD clause #15810

Closed

findepi requested review from electrum, martint, jklamer, losipiuk, sopel39, alexjo2144, anjalinorwood and ebyhr January 25, 2023 15:15

This was referenced Jan 25, 2023

Small code fixes #15843

Merged

Refactor materialized view tests #15844

Merged

findepi force-pushed the findepi/grace-iceberg branch from 676073f to 27e300e Compare January 25, 2023 15:20

github-actions bot added the docs label Jan 25, 2023

findepi force-pushed the findepi/grace-iceberg branch from 27e300e to 84ad01d Compare January 26, 2023 13:45

findepi mentioned this pull request Jan 26, 2023

Remove outdated SPI revapi exclusions #15863

Merged

findepi force-pushed the findepi/grace-iceberg branch 2 times, most recently from fbfaeb4 to d227435 Compare January 27, 2023 08:56

findepi force-pushed the findepi/grace-iceberg branch 5 times, most recently from d3090b3 to 1166c44 Compare January 31, 2023 11:03

findepi mentioned this pull request Feb 1, 2023

Fix materialized view failure mode when base table gone #15929

Merged

findepi force-pushed the findepi/grace-iceberg branch from 1166c44 to 62801b9 Compare February 1, 2023 13:52

This comment was marked as outdated.

Sign in to view

github-actions bot added the tests:hive label Feb 1, 2023

findepi force-pushed the findepi/grace-iceberg branch from 62801b9 to 5c781a7 Compare February 1, 2023 21:22

findepi force-pushed the findepi/grace-iceberg branch from ae7c300 to 558d586 Compare March 9, 2023 15:50

github-actions bot added the iceberg Iceberg connector label Mar 9, 2023