unify timestamping interval into a single system parameter#35027
unify timestamping interval into a single system parameter#35027petrosagg merged 8 commits intoMaterializeInc:mainfrom
Conversation
Pre-merge checklist
|
petrosagg
force-pushed
the
timestamp-interval-system-var
branch
2 times, most recently
from
2414c0f to
2ca8b31
Compare
def-
left a comment
def-
left a comment
There was a problem hiding this comment.
Nightly triggered: https://buildkite.com/materialize/nightly/builds/15175
I also want to see what happens if you lower it to 10 ms in all of CI, maybe something falls over in an interesting way: https://buildkite.com/materialize/test/builds/115951
| let min = scx.catalog.system_vars().min_timestamp_interval(); | ||
| let max = scx.catalog.system_vars().max_timestamp_interval(); |
There was a problem hiding this comment.
What happens if the min/max timestamp interval changes to a value such that duration is not within the bounds anymore? This will not cause the effective interval to be fixed such that it'll be within the interval, right? Maybe you can add an SLT test to show behavior if this happens. (I don't think we specified the behavior, so it might be a good idea to note it somewhere.)
There was a problem hiding this comment.
Hm, I didn't test it but the behavior we want is the same one we have for feature flags that allow you to create certain type of objects or use certain parameters, which is that those flags are marked with enable_for_item_parsing so that if you enable the flag, create the restricted object, then disable the flag the catalog can still be loaded.
I'll add a test for that
antiguru
left a comment
antiguru
left a comment
There was a problem hiding this comment.
Looks great in general, just one Q around adjusting the min/max intervals and what happens if custom durations fall outside the interval.
This PR replaces five dyncfg parameters (`pg_offset_known_interval`, `mysql_offset_known_interval`, `kafka_default_metadata_fetch_interval`, `sql_server_cdc_poll_interval`, `sql_server_offset_known_interval`) and a hardcoded 1s constant with a single `timestamp_interval` system parameter. This parameter controls the timestamping interval of tables and the default timestamping interval of sources. For sources, users can specify a custom timestamping interval as part of the `CREATE SOURCE` statement using the `TIMESTAMP INTERVAL` WITH option. When this parameter is not specified the source is created with the current value of the `timestamp_interval` system parameter. Changing the global `timestamp_interval` parameter won't affect already created sources. To change the timestamp interval of an already created source an `ALTER SOURCE` statement is needed. The value of `timestamp_interval` is validated against `min_timestamp_interval` and `max_timestamp_interval`. Signed-off-by: Petros Angelatos <petrosagg@gmail.com>
petrosagg
force-pushed
the
timestamp-interval-system-var
branch
from
d5d08b1 to
06770b8
Compare
kay-kim
left a comment
kay-kim
left a comment
There was a problem hiding this comment.
the docs changes lgtm. Left a question w.r.t. checking after someone issues the command.
|
|
||
| ### Changing the timestamp interval | ||
|
|
||
| To set a custom timestamp interval for a source: |
There was a problem hiding this comment.
Q: Is there a way to show the current timestamp interval for a source after someone sets this to verify?
4 participants
This PR replaces five dyncfg parameters (
pg_offset_known_interval,mysql_offset_known_interval,kafka_default_metadata_fetch_interval,sql_server_cdc_poll_interval,sql_server_offset_known_interval) and a hardcoded 1s constant with a singledefault_timestamp_intervalsystem parameter.This parameter controls the timestamping interval of tables and the default timestamping interval of sources. For sources, users can specify a custom timestamping interval as part of the
CREATE SOURCEstatement using theTIMESTAMP INTERVALWITH option. When this parameter is not specified the source is created with the current value of thedefault_timestamp_intervalsystem parameter. Changing the globaldefault_timestamp_intervalparameter won't affect already created sources. To change the timestamp interval of an already created source anALTER SOURCEstatement is needed.The value of the
TIMESTAMP INTERVALWITH option is validated against themin_timestamp_intervalandmax_timestamp_intervalsystem parameters.part of CLU-25