Add OffsetsForTimes API to Consumer #235

denisivan0v · 2017-06-20T06:24:04Z

Closes #228.

ghost · 2017-06-20T06:24:06Z

It looks like @denisivan0v hasn't signed our Contributor License Agreement, yet.

The purpose of a CLA is to ensure that the guardian of a project's outputs has the necessary ownership or grants of rights over all contributions to allow them to distribute under the chosen licence.
Wikipedia

You can read and sign our full Contributor License Agreement here.

Once you've signed reply with [clabot:check] to prove it.

Appreciation of efforts,

clabot

denisivan0v · 2017-06-20T06:52:55Z

[clabot:check]

ghost · 2017-06-20T06:52:56Z

It looks like @denisivan0v hasn't signed our Contributor License Agreement, yet.

The purpose of a CLA is to ensure that the guardian of a project's outputs has the necessary ownership or grants of rights over all contributions to allow them to distribute under the chosen licence.
Wikipedia

You can read and sign our full Contributor License Agreement here.

Once you've signed reply with [clabot:check] to prove it.

Appreciation of efforts,

clabot

denisivan0v · 2017-06-20T06:55:50Z

[clabot:check]

ghost · 2017-06-20T06:55:51Z

@confluentinc It looks like @denisivan0v just signed our Contributor License Agreement. 👍

Always at your service,

clabot

denisivan0v · 2017-06-20T06:59:45Z

@mhowlett Is there any feed where I can get these artifacts https://ci.appveyor.com/project/mhowlett/confluent-kafka-dotnet/build/1.0.53/artifacts ?

treziac · 2017-06-20T13:32:55Z

Thanks a lat for PR!

Not online (only release packages are pushed on nuget), but you can download them and add a repository as feed source to test them.
Will review later.

mhowlett · 2017-06-20T16:59:12Z

that was quick! thanks!

General comments:

We'll need API docs on public methods
We'll need an integration test to demonstrate it works in some basic way.

API choice:

I wouldn't use IReadOnlyDictionary. If we go with something compatible with dictionaries, I'd use IEnumerable<KeyValuePair< ... >> instead which is less specialized.

All of the following is up for discussion/change, just thinking out loud here:

Currently, no other method on Consumer (eg Position) accepts/returns dictionary compatible types. Use of IEnumerable<KeyValuePair<>> is quite natural, but for consistency with other methods, I'd say we should not use dictionaries for this right now. If we change to dictionaries, that is a future enhancement made across the entire library.

I'd invent a new type TopicPartitionTimestamp, where Timestamp should (definitely) be Confluent.Kafka's Timestamp, not DateTime. Input would be IEnumerable<TopicPartitionTimestamp>.

For return value I'd use IEnumerable<TopicPartitionOffset> (consistent with other methods). We use concrete List<..>s elsewhere, but we should definitely not be returning concrete types, this is a mistake. We should update other methods too, but not necessary in this PR.

I see in the returned collection you're including the Timestamp. Denormalized output is arguably better here, but it's nothing you couldn't easily do with a line of LINQ and also we don't do this anywhere else in the API, so I wouldn't do it here either. Again, discussion on this point encouraged!

Note to self: Read this closely, and review entire API:
https://docs.microsoft.com/en-us/dotnet/standard/design-guidelines/guidelines-for-collections

mhowlett · 2017-06-20T17:05:41Z

Also, I'm a fan of using TimeSpan for specifying timeouts as I think it makes code more readable. I'd use that instead of millisecondsTimeout.

denisivan0v · 2017-06-21T05:58:45Z

Thanks for your comments! I've made required changes to unify API and inserted API docs on public methods and types.

As for IEnumerable<TopicPartitionTimestamp> instead of IReadOnlyDictionary<>, this opens an ability to pass duplicates of TopicPartitionTimestamp and we'll get an exception while librdkafka call.

Also, I think that is better to use IReadOnlyCollection<> instead of IEnumerable<> because we always have a finite sequence of some instances in memory.

I'm going to also add integration test(s) later today as well.

…nc#235)

mhowlett · 2017-06-21T16:16:38Z

excellent! looking good at a quick glance - look forward to the integration test :-)

I don't think librdkafka will produce an error for duplicate input (or at least if I had written it it wouldn't..).

I think you can feel confident using this branch in your own code (most conveniently via the artifact produced on appveyor). I'm open to changing my mind on our use of our collections but I think for now consistency should win and I'll open a general issue soliciting feedback across the entire API.

FYI: the timeline for getting this merged is as follows:

I'm going on vacation for a few weeks from tomorrow, we'll get it merged to master soon after.
The 0.11.0 release is frozen unfortunately, we've missed that release.
we plan to set up more frequent pushes of preview snapshots to nuget.org, so it'll get in the first of those.

treziac · 2017-06-26T16:25:23Z

src/Confluent.Kafka/Producer.cs

@@ -1005,8 +1005,12 @@ public class Producer<TKey, TValue> : ISerializingProducer<TKey, TValue>, IDispo
            ISerializer<TValue> valueSerializer,
            bool manualPoll, bool disableDeliveryReports)
        {
-            var configWithoutKeySerializerProperties = KeySerializer.Configure(config, true);
-            var configWithoutValueSerializerProperties = ValueSerializer.Configure(config, false);
+            var configWithoutKeySerializerProperties = keySerializer != null


Yep, noticed it this we, it was introduced in #205 but waiting for matt to come back as there are other things to fix on it. This will be done in separate PR I think

That's ok if it will be another PR for this. But I need it to be fixed here as I'm going to use this CI build. I'll resolve conflicts if any.

treziac · 2017-06-26T16:27:29Z

Still didn't get proper time to review/try it, but should definitely have some this week ;)

denisivan0v · 2017-06-27T17:14:39Z

@mhowlett I've added an integration test and basic unit tests.

Also, I've found that most of the existing integration tests don't work (at least in my env).

Moreover, Producer<K,V> still hangs on ProduceAsync if K is Null. I'd spent several hours trying to fix it, but had no luck. There is no callback from librdkafka in this case.

denisivan0v · 2017-06-27T17:15:35Z

@treziac Looking forward to your review as well. Thanks!

treziac

Thanks again!

Major problem for me is the wrong values returned without exception, but it's more a librdkafka issue I think.

You should add integration test on what happen when using wrong timestamp (in future or too much in past) and check we properly throw.

Still have to review and test properly for possible issue

treziac · 2017-06-30T09:52:23Z

src/Confluent.Kafka/Impl/LibRdKafka.cs

@@ -713,6 +719,11 @@ private class NativeMethods
                    int partition, out long low, out long high);

            [DllImport(DllName, CallingConvention = CallingConvention.Cdecl)]
+            internal static extern ErrorCode rd_kafka_offsets_for_times(IntPtr rk,
+                /* rd_kafka_topic_partition_list_t * */ IntPtr topics,


offsets, not topics

treziac · 2017-06-30T09:52:40Z

src/Confluent.Kafka/Consumer.cs

@@ -612,6 +612,31 @@ public WatermarkOffsets QueryWatermarkOffsets(TopicPartition topicPartition)
            => consumer.QueryWatermarkOffsets(topicPartition);

        /// <summary>
+        ///     Look up the offsets for the given partitions by timestamp. The returned offset for each partition is the
+        ///     earliest offset whose timestamp is greater than or equal to the given timestamp in the corresponding partition.
+        ///


Remove empty line for consistency

Also, as you repeat the doc, you may want to use include_docs.xml (matt used it recently, this seems a good idea to use it when doc is duplicated)

So just need a
/// <include file='include_docs.xml' path='API/Member[@name="Consumer_OffsetsForTimes"]/*' />

And add the proper doc in the file

treziac · 2017-06-30T09:54:43Z

src/Confluent.Kafka/Impl/LibRdKafka.cs

@@ -391,6 +392,11 @@ internal static SafeTopicHandle topic_new(IntPtr rk, string topic, IntPtr conf)
                out long low, out long high)
            => _get_watermark_offsets(rk, topic, partition, out low, out high);

+        private delegate ErrorCode OffsetsForTimes(IntPtr rk, IntPtr topics, IntPtr timeout_ms);


offsets, not topics (by reading rdkafka.h, may change this comment later finaly)

treziac · 2017-06-30T10:29:19Z

src/Confluent.Kafka/Impl/SafeKafkaHandle.cs

+        internal IEnumerable<TopicPartitionOffset> OffsetsForTimes(IEnumerable<TopicPartitionTimestamp> timestampsToSearch, int millisecondsTimeout)
+        {
+            IEnumerable<TopicPartitionOffset> offsets = timestampsToSearch
+                .Select(t => new TopicPartitionOffset(t.TopicPartition, t.Timestamp.UnixTimestampMs))


This seems so weird with the TopicPartitionOffset taking a long timestamp (though required by librdkafka)
Add some commentary to explain quickly why this is done, or perhaps create a new GetCTopicPartitionList(IEnumerable<TopicPartitionTimestamp> offsets) - not sure what is best

Added a comment with explanation

treziac · 2017-06-30T11:57:37Z

test/Confluent.Kafka.IntegrationTests/Tests/Tests.cs

@@ -30,13 +30,19 @@ public static IEnumerable<object[]> KafkaParameters()
        {
            if (kafkaParameters == null)
            {
-                var codeBase = typeof(Tests).GetTypeInfo().Assembly.CodeBase;


Perhaps in another PR - I never took time to do it for now as I wanted to put properly appveyor for integration tests

Assembly.Location provides the full path of the tests dll. And it's exactly what is needed here.

From the MSDN article for the CodeBase property:

To get the absolute path to the loaded manifest-containing file, use the Assembly.Location property instead.

I can rollback this and move to anther PR. Should I do it?

treziac · 2017-06-30T11:59:04Z

test/Confluent.Kafka.IntegrationTests/Tests/Tests.cs

                var assemblyDirectory = Path.GetDirectoryName(assemblyPath);
                var jsonPath = Path.Combine(assemblyDirectory, "kafka.parameters.json");
-                dynamic json = JsonConvert.DeserializeObject(File.ReadAllText(jsonPath));
-                kafkaParameters = new List<object[]>() { new object[] { json.bootstrapServers.ToString(), json.topic.ToString(), json.partitionedTopic.ToString() } };
+                var json = JObject.Parse(File.ReadAllText(jsonPath));


Any reason to go from JsonConvert.DeserializeObject to JObject.Parse?
Same things, would be better in another PR with the previous one

With JsonConvert.DeserializeObject we need to have strongly typed object (or dynamic object, like in this case). With JObject.Parse we have parsed JSON tree as the result and can access to the object properties without any other object creations. In my opinion, it's better to use JObject here instead of dynamic object.

treziac · 2017-06-30T12:01:04Z

test/Confluent.Kafka.IntegrationTests/Tests/Consumer_OffsetsForTimes.cs

+                // See librdkafka implementation for details https://github.com/edenhill/librdkafka/blob/master/src/rdkafka.c#L2433
+                var result = consumer.OffsetsForTimes(
+                        new[] {new TopicPartitionTimestamp(messages[0].TopicPartition, messages[0].Timestamp)},
+                        TimeSpan.FromSeconds(10))


I have have some issues here, I think it is librdkafka related
When passing TimeSpan.FromSeconds(1), I don't receive any error but the offset is not updated - I retrieve timestamp as offset. Will try to test more this we - won't have more time for reviewing today

That's correct. The current rdlibkafka implementation just returns from the wait loop when timeout ends. This leads to the issue when offset is not updated, unfortunately.
See https://github.com/edenhill/librdkafka/blob/master/src/rdkafka.c#L2475 for details

treziac · 2017-06-30T12:02:37Z

test/Confluent.Kafka.IntegrationTests/Tests/Consumer_OffsetsForTimes.cs

+            {
+                for (int index = 0; index < N; index++)
+                {
+                    var message = await producer.ProduceAsync(topic, $"test key {index}", $"test val {index}");


To avoid having too long tests, prefer using an IDeliveryHandler, allowing you to produce all your messages async - this is not a no-go, feel free to change it or not.

Tried using IDeliveryHandler to make code shorter, but this approach gives me about the same. I'd like to leave it like this, if you don't mind.

treziac · 2017-06-30T12:22:30Z

src/Confluent.Kafka/TopicPartitionTimestamp.cs

+        /// <param name="timestamp">
+        ///     A Kafka timestamp value.
+        /// </param>
+        public TopicPartitionTimestamp(string topic, int partition, Timestamp timestamp)


We may modify Timestamp back to the way where it accepted Datetime - if we want to retrieve offset at a particular date, it's kind of heavy with current api (new Timestamp(Timestamp.FromDateTime(...)), or perhaps just add a Timestamp.FromDateTime static method, or an implicit/explicit conversion from datetime to timestamp.

Added Timestamp.FromDateTime method

I agree that we want to provide easier conversion from DateTime to Timestamp.

Of the options @treziac notes, I'm definitely against implicit conversion and TimestampType needs to be specified so explicit conversion is out too.

Between the static method and constructor, I'm currently favoring using a constructor because 1. it's fewer characters. 2. we only ever want to return a Timestamp struct - we don't need the flexibility provided by a static method. 3. It's slightly incorrectly named - DateTime is not the only parameter. 4. the constructor variant reads easier to me.

I'm definitely open to being convinced back to using a static method if someone wants to present a good argument, but currently think we should change the FromDateTime method to an additional Timestamp constructor.

For reference, some previous conversation related to DateTime/Timestamps is in #120

Totally agreed that a constructor much better than the static method. Done in a517357

treziac

Edit: found the error by proper checking of librdkafka code. Need to adapt your code and add tests given this scenario

treziac · 2017-06-30T13:08:23Z

src/Confluent.Kafka/Impl/SafeKafkaHandle.cs

+
+            var list = GetTopicPartitionOffsetErrorList(cOffsets);
+            LibRdKafka.topic_partition_list_destroy(cOffsets);
+            return list.Select(t => t.TopicPartitionOffset).ToList();


You can have error by partition - the error returned by offsets_for_times i a general error (broker not available), but each partition may return an error. In this case, the offset will still be at the timestamp. You can reproduce easily by querying to an inexistant partition.

Two choices: either we return a list of TopicPartitionOffsetError (and user will have to map it before assign), either we throw exception if any member has error. I prefer first solution, eventualy with an implcit cast from TopicPartitionErrorOffset to topicPartitionOffset which would throw if an error is present

Not sure that I can repro the partition error when querying non-existent partition. In my case I've got a general error.

But I've made checks for errors for every partition, and in case of any error OffsetsForTimes throws the KafkaException for the first found error.

Thoughts?

I prefer first solution,

yes, I agree (as noted elsewhere).

eventualy with an implcit cast from TopicPartitionErrorOffset to topicPartitionOffset which would throw if an error is present

This is an interesting (good) idea. We could do implicit or explicit casting or have a ToTopicPartitionOffset method. A potential problem is confusion with the TopicPartitionOffset property (which doesn't throw). We could make that throw, but then that would be inconsistent with other properties.

It's intuitive to me that a cast would throw if there was a problem, so I like the idea of using a cast for conversion (though it could be argued the semantics aren't perfectly clear). We want an explicit cast though, as the operation can fail. See:

https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/types/casting-and-type-conversions

I think we should include that in this PR.

Personally, I'm not a big fan of cast operations that can throw. Client can test Error property before get a TopicPartitionOffset without cast operation and try/catch block.

However, it was easy to add an explicit cast operation and unit test for it, so I did it. Thanks!

denisivan0v · 2017-07-13T09:20:33Z

@treziac many thanks for your very detailed review. Please check the changes I've made.

I started to use the CI build in my project and will report other issues if any.

mhowlett · 2017-07-21T23:50:53Z

src/Confluent.Kafka/Impl/SafeKafkaHandle.cs

+                }
+
+                var list = GetTopicPartitionOffsetErrorList(cOffsets);
+                var error = list.Where(x => x.Error.Code != ErrorCode.NoError).Select(x => x.Error).FirstOrDefault();


When I suggested making a TopicPartitionTimestamp type, I wasn't realizing that offsets_for_times could return per partition errors. Sorry! We should not have the TopicPartitionTimestamp type, replace it with a TopicPartitionTimestampError type and not throw an exception when a partition specific error occurs.

Oh sorry, we return TopicPartitionOffset not TopicPartitionTimestamp. So keep the TopicPartitionTimestamp type but return TopicPartitionOffsetError instead of TopicPartitionOffset (and get rid of the check for per partition errors here).

Done in a517357, thanks!

mhowlett · 2017-07-22T15:35:25Z

test/Confluent.Kafka.UnitTests/Confluent.Kafka.UnitTests.csproj

@@ -16,4 +16,8 @@
    <PackageReference Include="xunit.runner.visualstudio" Version="2.2.0" />
  </ItemGroup>

+  <ItemGroup>


don't think we want this ItemGroup

Visual studio keeps adding it, I think we could include it. Didn't look if they changed it in 2.0

I think we can do this https://stackoverflow.com/a/33414270

Done in a517357

mhowlett · 2017-07-24T23:17:48Z

thanks for the latest commits @denisivan0v - this is looking good to me now. Will review again tomorrow and probably merge :-)

denisivan0v · 2017-07-25T04:54:13Z

Many thanks for your changes and upstream merge @mhowlett. Looking forward for any updates here.

denisivan0v added 2 commits June 20, 2017 00:46

WIP: Implementing Consumer.OffsetsForTimes. Interacting with librdkafka

8ff96ae

confluentinc#228 Consumer.OffsetsForTimes implemented

112b286

Changes on review comments (confluentinc#235)

62a0421

Changes for timeout param on Consumer.OffsetsForTimes API (confluenti…

a4158e5

…nc#235)

Bugfix: null checks in Producer ctor

1089287

treziac reviewed Jun 26, 2017

View reviewed changes

denisivan0v added 2 commits June 27, 2017 23:43

Intergration test for Consumer.OffsetsForTimes added (confluentinc#235)

b4f54dd

Unit tests added for TopicPartitionTimestamp (confluentinc#235)

8b4c360

treziac reviewed Jun 30, 2017

View reviewed changes

treziac suggested changes Jun 30, 2017

View reviewed changes

denisivan0v added 3 commits July 13, 2017 15:31

changes on codereview (confluentinc#235)

34c1ca0

updated from upstream

f366588

argument renaming in rd_kafka_offsets_for_times (confluentinc#235)

938aca2

mhowlett added 2 commits July 21, 2017 16:22

Merge branch 'master' into master

ba83138

fixed incorrect merge

52bd0a4

removed space

9293799

mhowlett reviewed Jul 21, 2017

View reviewed changes

mhowlett reviewed Jul 22, 2017

View reviewed changes

denisivan0v and others added 3 commits July 24, 2017 23:57

changes on codereview (confluentinc#235)

a517357

small code cleaning in tests

c930399

minor changes

14dbe3a

mhowlett merged commit 3b7bde7 into confluentinc:master Jul 25, 2017

Add OffsetsForTimes API to Consumer #235

Add OffsetsForTimes API to Consumer #235

Conversation

denisivan0v commented Jun 20, 2017

ghost commented Jun 20, 2017

denisivan0v commented Jun 20, 2017 • edited Loading

ghost commented Jun 20, 2017

denisivan0v commented Jun 20, 2017

ghost commented Jun 20, 2017

denisivan0v commented Jun 20, 2017

treziac commented Jun 20, 2017

mhowlett commented Jun 20, 2017

mhowlett commented Jun 20, 2017

denisivan0v commented Jun 21, 2017

mhowlett commented Jun 21, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

treziac commented Jun 26, 2017

denisivan0v commented Jun 27, 2017

denisivan0v commented Jun 27, 2017

treziac left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

denisivan0v Jul 13, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

denisivan0v Jul 13, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

denisivan0v Jul 13, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

denisivan0v Jul 24, 2017 • edited Loading

Choose a reason for hiding this comment

treziac left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

denisivan0v commented Jul 13, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mhowlett commented Jul 24, 2017

denisivan0v commented Jul 25, 2017

denisivan0v commented Jun 20, 2017 •

edited

Loading

denisivan0v Jul 13, 2017 •

edited

Loading

denisivan0v Jul 13, 2017 •

edited

Loading

denisivan0v Jul 13, 2017 •

edited

Loading

denisivan0v Jul 24, 2017 •

edited

Loading