KAFKA-6521: Use timestamped stores for KTables #6667

mjsax · 2019-05-02T21:20:01Z

The tests are failing because #6661 is not merged yet..

The goal of this PR is to swap out, KeyValueStore and WindowStore with the corresponding new TimestampedXxxStore in the DSL where appropriate. We don't leverage the timestamps yet in the Processors, to keep the PR contained. Thus, no semantic change yet.

mjsax · 2019-05-02T21:21:17Z

Call for early review @guozhangwang @bbejeck @vvcephei @ableegoldman @cadonna @abbccdda

mjsax · 2019-05-02T21:22:22Z

.../src/main/java/org/apache/kafka/streams/state/internals/MeteredTimestampedKeyValueStore.java

@@ -51,6 +51,6 @@ void initStoreSerde(final ProcessorContext context) {
        serdes = new StateSerdes<>(
            ProcessorStateManager.storeChangelogTopic(context.applicationId(), name()),
            keySerde == null ? (Serde<K>) context.keySerde() : keySerde,
-            valueSerde == null ? new ValueAndTimestampSerde<>((Serde<V>) context.keySerde()) : valueSerde);
+            valueSerde == null ? new ValueAndTimestampSerde<>((Serde<V>) context.valueSerde()) : valueSerde);


Fix bug that was exposed with this PR.

Oof. Good catch!

vvcephei

Thanks for the PR, @mjsax . I guess the idea here is to just mix in the new (timestamped) stores without changing any semantics? And then we'll have other PR(s) later to update the semantics and tests? This seems like a good way to break it down.

Just a few comments.

vvcephei · 2019-05-06T20:22:39Z

...n/java/org/apache/kafka/streams/kstream/internals/KTableMaterializedValueGetterSupplier.java

        }

        @Override
        public V get(final K key) {
-            return store.get(key);
+            final ValueAndTimestamp<V> valueAndTimestamp = store.get(key);
+            return valueAndTimestamp == null ? null : valueAndTimestamp.value();


This logic has come up once or twice ;)

Just a random idea... What do you think about a static method on ValueAndTimestamp:

static <V> V getValueOrNull(final ValueAndTimestamp<V> valueAndTimestamp) { return valueAndTimestamp == null ? null : valueAndTimestamp.value(); }

Then all these other methods could be:

Suggested change

return valueAndTimestamp == null ? null : valueAndTimestamp.value();

return getValueOrNull(valueAndTimestamp);

?

Ack. I'll just update the KIP accordingly. Turn out, we already have ValueAndTimestamp#make and no public constructor (also not documented on the KIP).

vvcephei · 2019-05-06T20:39:37Z

.../src/main/java/org/apache/kafka/streams/state/internals/MeteredTimestampedKeyValueStore.java

@@ -51,6 +51,6 @@ void initStoreSerde(final ProcessorContext context) {
        serdes = new StateSerdes<>(
            ProcessorStateManager.storeChangelogTopic(context.applicationId(), name()),
            keySerde == null ? (Serde<K>) context.keySerde() : keySerde,
-            valueSerde == null ? new ValueAndTimestampSerde<>((Serde<V>) context.keySerde()) : valueSerde);
+            valueSerde == null ? new ValueAndTimestampSerde<>((Serde<V>) context.valueSerde()) : valueSerde);


Oof. Good catch!

vvcephei · 2019-05-06T20:41:31Z

...ms/src/main/java/org/apache/kafka/streams/state/internals/TimestampedWindowStoreBuilder.java

@@ -36,7 +36,7 @@ public TimestampedWindowStoreBuilder(final WindowBytesStoreSupplier storeSupplie
                                         final Serde<K> keySerde,
                                         final Serde<V> valueSerde,
                                         final Time time) {
-        super(storeSupplier.name(), keySerde, new ValueAndTimestampSerde<>(valueSerde), time);
+        super(storeSupplier.name(), keySerde, valueSerde == null ? null : new ValueAndTimestampSerde<>(valueSerde), time);


I found it useful with the FullChangeSerde to make the constructor private and force usage via a null-safe factory method. Then, you never have to worry about this check.

Seems to be a single occurrence. Maybe not worth a factory. Also, ValueAndTimestampSerde checks for non-null and thus we would surface any issue during testing anyway.

mjsax · 2019-05-06T23:35:05Z

Updated this.

vvcephei

Thanks for the update!

Just another minor comment on ValueAndTimestamp

vvcephei · 2019-05-07T15:02:03Z

streams/src/main/java/org/apache/kafka/streams/state/ValueAndTimestamp.java

@@ -29,8 +29,8 @@
    private final V value;
    private final long timestamp;

-    private ValueAndTimestamp(final V value,
-                              final long timestamp) {
+    protected ValueAndTimestamp(final V value,


Since the only usage of the constructor is in this class, it should be ok to be private.

Also, since it's just a data struct, we could also make the class final.

mjsax · 2019-05-07T15:09:56Z

Updated this.

abbccdda · 2019-05-07T16:42:06Z

streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamAggregate.java

 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;

+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;


We already import the class on L26, so let's remove this import.

This is for statically use this single util function (otherwise we have to call ValueAndTimestamp.getValueOrNull when calling).

abbccdda · 2019-05-07T16:42:56Z

streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamWindowAggregate.java

 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;

 import java.util.Map;

+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;


abbccdda · 2019-05-07T16:44:02Z

streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamWindowAggregate.java

        }

        @Override
-        public void close() {


Is having curly braces in one line a convention?

abbccdda · 2019-05-07T16:44:21Z

streams/src/main/java/org/apache/kafka/streams/kstream/internals/KTableFilter.java

@@ -117,6 +120,7 @@ public void process(final K key, final Change<V> change) {
        }
    }

+


Seems unnecessary L123

I like to have some more line to separate nested classes from method

abbccdda · 2019-05-07T16:46:07Z

streams/src/main/java/org/apache/kafka/streams/kstream/internals/TimestampedTupleForwarder.java

+ * Forwarding by this class only occurs when caching is not enabled. If caching is enabled,
+ * forwarding occurs in the flush listener when the cached store flushes.
+ *
+ * @param <K>


We could omit L28-29 since we don't want to comment on it.

Good eye. It would be even better to document the parameters!

abbccdda · 2019-05-07T16:47:18Z

streams/src/main/java/org/apache/kafka/streams/kstream/internals/TimestampedTupleForwarder.java

+    public void maybeForward(final K key,
+                             final V newValue,
+                             final V oldValue) {
+        if (cachingEnabled) {


How about

if (!cachingEnabled) { context.forward(key, new Change<>(newValue, oldValue)); }

abbccdda · 2019-05-07T16:50:06Z

streams/src/main/java/org/apache/kafka/streams/state/ValueAndTimestamp.java

+     *
+     * @param valueAndTimestamp a {@link ValueAndTimestamp} instance; can be {@code null}
+     * @param <V> the type of the value
+     * @return the wrapped {@code value} of {@code valueAndTimestamp} is not {@code null}; otherwise {@code null}


s/is not/if not

bbejeck · 2019-05-07T18:12:17Z

#6661 has been merged now.

retest this please

bbejeck

Thanks @mjsax LGTM

guozhangwang · 2019-05-07T22:31:27Z

retest this please

guozhangwang

Some minor comments.

guozhangwang · 2019-05-07T22:34:10Z

streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamAggregate.java

 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;

+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;


This is for statically use this single util function (otherwise we have to call ValueAndTimestamp.getValueOrNull when calling).

guozhangwang · 2019-05-07T22:39:40Z

streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamReduce.java

@@ -88,14 +95,13 @@ public void process(final K key, final V value) {
            }

            // update the store with the new value
-            store.put(key, newAgg);
+            store.put(key, ValueAndTimestamp.make(newAgg, context().timestamp()));


Today we store the timestamp in cache entry and upon flushing we call entry.entry().context().timestamp() to set the timestamp. Would that be replaced with the underlying store's timestamp then?

Maybe. After the store refactoring, I realized that we might actually be able to revert #6147 -- however, I would prefer to do this as cleanup only after everything else is merged (to make sure it's really safe).

While working on a follow PR, I figure out the details. The TimestampedForwardingCacheFlushListener needs to extract the timestamp from the value, and set it as record timestamp. Similar to SessionCacheFlushListener in another PR: https://github.com/apache/kafka/pull/6645/files#diff-4039efe54e3b87d65c20bd7d14c1b149R43

guozhangwang · 2019-05-07T22:42:12Z

streams/src/main/java/org/apache/kafka/streams/kstream/internals/KTableTransformValues.java

@@ -105,8 +109,13 @@ public void process(final K key, final Change<V> change) {
                final V1 oldValue = sendOldValues ? valueTransformer.transform(key, change.oldValue) : null;
                context().forward(key, new Change<>(newValue, oldValue));
            } else {
-                final V1 oldValue = sendOldValues ? store.get(key) : null;
-                store.put(key, newValue);
+                final V1 oldValue;


nit: we can still use the ternary operator here.

guozhangwang · 2019-05-07T22:44:09Z

...java/org/apache/kafka/streams/kstream/internals/TimestampedForwardingCacheFlushListener.java

+
+import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;
+
+class TimestampedForwardingCacheFlushListener<K, V> implements CacheFlushListener<K, ValueAndTimestamp<V>> {


Can we just remove the old CacheFlushListener / TupleForwarder since they are internal classes only used in the DSL?

We still need the old one, because we don't use TimestampedSessionWindowStore in the DSL but keep SessionWindowStore.

guozhangwang · 2019-05-07T22:47:40Z

streams/src/test/java/org/apache/kafka/streams/kstream/internals/KTableReduceTest.java

-import org.apache.kafka.test.GenericInMemoryKeyValueStore;
+import org.apache.kafka.streams.state.TimestampedKeyValueStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.apache.kafka.test.GenericInMemoryTimestampedKeyValueStore;


Can we replace other uses of GenericInMemoryKeyValueStore so we can remove the old class as well?

Not sure. We don't remove "plain" stores, and thus it seems to make sense to keep existing tests and only rewrite DSL related tests?

mjsax · 2019-05-08T14:48:35Z

Updated and rebased this.

mjsax · 2019-05-08T14:57:22Z

.../src/main/java/org/apache/kafka/streams/state/internals/TimestampedKeyValueStoreBuilder.java

+                store = new KeyValueToTimestampedKeyValueByteStoreAdapter(store);
+            } else {
+                store = new InMemoryTimestampedKeyValueStoreMarker(store);
+            }


This fixes a bug that was exposed after rebasing. We need to treat in-memory stores always as TimestampedByteStores to guarantee correct restore. (Similar below for window stores.)

Added a new test to #6179 that also exposed this bug (cf. StoreUpgradeIntegrationTest)

I agree with the fix, but it raises another question for me.

I admit I could be either missing something or forgetting some details already spelled out. Why don't we convert the values for in-memory stores to also have timestamps in the value? I'm basing on the fact that the InMemoryTimestampedKeyValueStoreMarker does simple puts/gets.

The in-memory store will have no content on startup, and the upper layer will put/get byte[] -- the store does not care what is in it. The upper layer will already serialize value+timestamp and put those, and expects value+timestamp byte[] back -- hence, the in-memory store does not need to do any translation.

For persistent store, the store does have content on local disk, namely plain values. Because this store cannot be upgraded, we put the adapter in between, to remove the timestamp part on put() and add dummy timestamp on get(), to meet the contract to the upper layer that expects byte[] to be value+timestamp.

If we restore a persistent store, the just copy plain key/value from the changelog topic and put it into the store. Everything still works as expected.

If we restore an in-memory store, we need to add the timestamp to the value, and this is triggered by the TimestampeByteStore interface. Otherwise, the in-memory store would contain plain values, and violate the contract to the upper layer, because the assumption is, that everything store in it, has value+timestamp format.

Ie, for persistent store, we play the backward compatibility game using surrogate timestamps. However, for in-memory store, we don't need to do this, because there is no local on-disk content and we can switch to timestamped-store behavior without any drawbacks and get full feature set.

Does this make sense?

yep thanks for clarifying

mjsax · 2019-05-08T14:59:57Z

streams/src/test/java/org/apache/kafka/streams/processor/internals/StandbyTaskTest.java

@@ -319,7 +321,7 @@ public void shouldRestoreToWindowedStores() throws IOException {
            changelogName,
            1,
            offset,
-            start,
+            end,


"exposed" after rebase -- for time-windows, we want to use the end-timestamp (even if this change is not implemented yet, I added it here, to avoid touching this class later on again).

mjsax · 2019-05-08T15:01:18Z

.../test/java/org/apache/kafka/streams/state/internals/TimestampedKeyValueStoreBuilderTest.java

@@ -46,14 +44,16 @@
    @Mock(type = MockType.NICE)
    private KeyValueBytesStoreSupplier supplier;
    @Mock(type = MockType.NICE)
-    private KeyValueStore<Bytes, byte[]> inner;
+    private RocksDBTimestampedStore inner;


The new wrapping would break the test. Using RocksDBTimestampedStore avoid the wrapping as TimestampedByteStore interface is implemented. (Same below.)

mjsax · 2019-05-08T22:42:40Z

Rebased to resolve merge conflicts.

mjsax · 2019-05-11T12:05:36Z

...ams/src/main/java/org/apache/kafka/streams/kstream/internals/SessionWindowedKStreamImpl.java

@@ -193,7 +194,7 @@
            }
            supplier = Stores.persistentSessionStore(
                materialized.storeName(),
-                retentionPeriod
+                Duration.ofMillis(retentionPeriod)


Minor fix on-the-side to not use deprecated method any longer.

mjsax · 2019-05-11T12:06:29Z

.../src/main/java/org/apache/kafka/streams/kstream/internals/TimestampedCacheFlushListener.java

+            context.forward(
+                key,
+                new Change<>(getValueOrNull(newValue), getValueOrNull(oldValue)),
+                To.all().withTimestamp(newValue != null ? newValue.timestamp() : timestamp));


New fix: On flush, we take the timestamp from the newValue if it exists.

mjsax · 2019-05-11T12:07:21Z

streams/src/main/java/org/apache/kafka/streams/kstream/internals/TupleForwarder.java

        this.context = context;
+        this.sendOldValues = sendOldValues;


New fix: TupleForwarder should obey sendOldValues, too.

mjsax · 2019-05-11T12:08:13Z

streams/src/main/java/org/apache/kafka/streams/state/internals/CachingKeyValueStore.java


-class CachingKeyValueStore
+public class CachingKeyValueStore


For testing...

mjsax · 2019-05-11T12:11:37Z

...in/java/org/apache/kafka/streams/kstream/internals/TimestampedKeyValueStoreMaterializer.java

        KeyValueBytesStoreSupplier supplier = (KeyValueBytesStoreSupplier) materialized.storeSupplier();
        if (supplier == null) {
            final String name = materialized.storeName();
-            supplier = Stores.persistentKeyValueStore(name);
+            supplier = Stores.persistentTimestampedKeyValueStore(name);


New: slipped in the first commit of this PR... discovered while working on a follow up PR.

mjsax · 2019-05-11T12:14:11Z

Rebased to resolve merge conflict. Also added some more fixes and tests.

guozhangwang · 2019-05-11T18:33:48Z

LGTM. Please feel free to merge after green builds.

guozhangwang · 2019-05-11T18:33:54Z

retest this please

…es-14-May * AK_REPO/trunk: (24 commits) KAFKA-7321: Add a Maximum Log Compaction Lag (KIP-354) (apache#6009) KAFKA-8335; Clean empty batches when sequence numbers are reused (apache#6715) KAFKA-6455: Session Aggregation should use window-end-time as record timestamp (apache#6645) KAFKA-6521: Use timestamped stores for KTables (apache#6667) [MINOR] Consolidate in-memory/rocksdb unit tests for window & session store (apache#6677) MINOR: Include StickyAssignor in system tests (apache#5223) KAFKA-7633: Allow Kafka Connect to access internal topics without cluster ACLs (apache#5918) MINOR: Align KTableAgg and KTableReduce (apache#6712) MINOR: Fix code section formatting in TROGDOR.md (apache#6720) MINOR: Remove unnecessary OptionParser#accepts method call from PreferredReplicaLeaderElectionCommand (apache#6710) KAFKA-8352 : Fix Connect System test failure 404 Not Found (apache#6713) KAFKA-8348: Fix KafkaStreams JavaDocs (apache#6707) MINOR: Add missing option for running vagrant-up.sh with AWS to vagrant/README.md KAFKA-8344; Fix vagrant-up.sh to work with AWS properly MINOR: docs typo in '--zookeeper myhost:2181--execute' MINOR: Remove header and key/value converter config value logging (apache#6660) KAFKA-8231: Expansion of ConnectClusterState interface (apache#6584) KAFKA-8324: Add close() method to RocksDBConfigSetter (apache#6697) KAFKA-6789; Handle retriable group errors in AdminClient API (apache#5578) KAFKA-8332: Refactor ImplicitLinkedHashSet to avoid losing ordering when converting to Scala ...

Reviewers: John Roesler <john@confluent.io>, Boyang Chen <boyang@confluent.io>, Bill Bejeck <bill@confluent.io>, Guozhang Wang <guozhang@confluent.io>

mjsax added the streams label May 2, 2019

mjsax commented May 2, 2019

View reviewed changes

vvcephei approved these changes May 6, 2019

View reviewed changes

vvcephei approved these changes May 7, 2019

View reviewed changes

abbccdda reviewed May 7, 2019

View reviewed changes

bbejeck approved these changes May 7, 2019

View reviewed changes

guozhangwang reviewed May 7, 2019

View reviewed changes

mjsax force-pushed the kafka-6521-add-timestamps-to-ktables branch from 7048ea9 to 161d6fc Compare May 8, 2019 14:48

mjsax commented May 8, 2019

View reviewed changes

mjsax force-pushed the kafka-6521-add-timestamps-to-ktables branch from 161d6fc to 73f05da Compare May 8, 2019 22:42

mjsax mentioned this pull request May 8, 2019

KAFAK-3522: Add TopologyTestDriver unit tests #6179

Merged

KAFKA-6521: Use timestamped stores for KTables

0e6f8d3

mjsax force-pushed the kafka-6521-add-timestamps-to-ktables branch from 73f05da to 92c0f6f Compare May 11, 2019 12:03

mjsax commented May 11, 2019

View reviewed changes

Follow up

21e5dee

mjsax force-pushed the kafka-6521-add-timestamps-to-ktables branch from b79b478 to 21e5dee Compare May 11, 2019 12:26

mjsax mentioned this pull request May 11, 2019

KAFKA-6455: Session Aggregation should use window-end-time as record timestamp #6645

Merged

mjsax merged commit 8649717 into apache:trunk May 12, 2019

mjsax deleted the kafka-6521-add-timestamps-to-ktables branch May 12, 2019 09:51

mjsax added the kip Requires or implements a KIP label Jun 12, 2020

	return valueAndTimestamp == null ? null : valueAndTimestamp.value();
	return getValueOrNull(valueAndTimestamp);

		@@ -117,6 +120,7 @@ public void process(final K key, final Change<V> change) {
		}
		}


		import static org.apache.kafka.streams.state.ValueAndTimestamp.getValueOrNull;

		class TimestampedForwardingCacheFlushListener<K, V> implements CacheFlushListener<K, ValueAndTimestamp<V>> {

KAFKA-6521: Use timestamped stores for KTables #6667

KAFKA-6521: Use timestamped stores for KTables #6667

Conversation

mjsax commented May 2, 2019 • edited Loading

mjsax commented May 2, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vvcephei left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mjsax commented May 6, 2019

vvcephei left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mjsax commented May 7, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bbejeck commented May 7, 2019

bbejeck left a comment

Choose a reason for hiding this comment

guozhangwang commented May 7, 2019

guozhangwang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mjsax commented May 8, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bbejeck May 9, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mjsax May 8, 2019 • edited Loading

Choose a reason for hiding this comment

mjsax commented May 8, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mjsax commented May 11, 2019

guozhangwang commented May 11, 2019

guozhangwang commented May 11, 2019

mjsax commented May 2, 2019 •

edited

Loading

mjsax commented May 2, 2019 •

edited

Loading

bbejeck May 9, 2019 •

edited

Loading

mjsax May 8, 2019 •

edited

Loading