Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Re-processing of records on restart because of conflicting offsets ?? #20

Closed
yatharthranjan opened this issue Jul 3, 2018 · 5 comments
Closed

Re-processing of records on restart because of conflicting offsets ?? #20

yatharthranjan opened this issue Jul 3, 2018 · 5 comments

Comments

@yatharthranjan
Copy link
Member

I get a lot of messages like following on restart of the container -

Fetch offset 24 is out of range for partition questionnaire_phq8-1, resetting offset
Resetting offset for partition questionnaire_phq8-1 to offset 11

This causes records to be processed again. Since the HDFS connector reads offsets from the file names, here is a snapshot of the files in hdfs for the above case.

hdfs dfs -ls /topicAndroidNew/questionnaire_phq8

drwxr-xr-x   - root supergroup          0 2018-04-26 14:28 /topicAndroidNew/questionnaire_phq8/partition=0
drwxr-xr-x   - root supergroup          0 2018-06-04 10:34 /topicAndroidNew/questionnaire_phq8/partition=0_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-1
drwxr-xr-x   - root supergroup          0 2018-06-20 11:01 /topicAndroidNew/questionnaire_phq8/partition=0_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-2
drwxr-xr-x   - root supergroup          0 2018-06-26 07:20 /topicAndroidNew/questionnaire_phq8/partition=0_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-4
drwxr-xr-x   - root supergroup          0 2018-07-03 09:49 /topicAndroidNew/questionnaire_phq8/partition=0_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-5
drwxr-xr-x   - root supergroup          0 2018-05-03 08:14 /topicAndroidNew/questionnaire_phq8/partition=1
drwxr-xr-x   - root supergroup          0 2018-06-25 09:35 /topicAndroidNew/questionnaire_phq8/partition=1_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-1
drwxr-xr-x   - root supergroup          0 2018-06-20 11:01 /topicAndroidNew/questionnaire_phq8/partition=1_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-2
drwxr-xr-x   - root supergroup          0 2018-06-20 13:10 /topicAndroidNew/questionnaire_phq8/partition=1_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-3
drwxr-xr-x   - root supergroup          0 2018-06-26 21:40 /topicAndroidNew/questionnaire_phq8/partition=1_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-4
drwxr-xr-x   - root supergroup          0 2018-07-02 12:14 /topicAndroidNew/questionnaire_phq8/partition=1_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-6
drwxr-xr-x   - root supergroup          0 2018-05-03 08:14 /topicAndroidNew/questionnaire_phq8/partition=2
drwxr-xr-x   - root supergroup          0 2018-05-31 19:47 /topicAndroidNew/questionnaire_phq8/partition=2_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-1
drwxr-xr-x   - root supergroup          0 2018-06-20 11:01 /topicAndroidNew/questionnaire_phq8/partition=2_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-2
drwxr-xr-x   - root supergroup          0 2018-06-20 15:09 /topicAndroidNew/questionnaire_phq8/partition=2_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-4
drwxr-xr-x   - root supergroup          0 2018-07-03 08:14 /topicAndroidNew/questionnaire_phq8/partition=2_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-5

hdfs dfs -ls /topicAndroidNew/questionnaire_phq8/partition=1

-rw-r--r--   3 root supergroup       4445 2018-03-16 11:28 /topicAndroidNew/questionnaire_phq8/partition=1/questionnaire_phq8+1+0000000000+0000000000.avro
-rw-r--r--   3 root supergroup       4445 2018-03-22 09:02 /topicAndroidNew/questionnaire_phq8/partition=1/questionnaire_phq8+1+0000000001+0000000001.avro
-rw-r--r--   3 root supergroup       4972 2018-04-26 04:55 /topicAndroidNew/questionnaire_phq8/partition=1/questionnaire_phq8+1+0000000001+0000000003.avro
-rw-r--r--   3 root supergroup       5237 2018-05-02 21:57 /topicAndroidNew/questionnaire_phq8/partition=1/questionnaire_phq8+1+0000000001+0000000004.avro
-rw-r--r--   3 root supergroup       5485 2018-05-03 08:14 /topicAndroidNew/questionnaire_phq8/partition=1/questionnaire_phq8+1+0000000001+0000000005.avro
-rw-r--r--   3 root supergroup       4459 2018-04-23 19:52 /topicAndroidNew/questionnaire_phq8/partition=1/questionnaire_phq8+1+0000000002+0000000002.avro
-rw-r--r--   3 root supergroup       4621 2018-03-22 09:17 /topicAndroidNew/questionnaire_phq8/partition=1/questionnaire_phq8+1+0000000002+0000000003.avro
-rw-r--r--   3 root supergroup       4462 2018-04-25 08:34 /topicAndroidNew/questionnaire_phq8/partition=1/questionnaire_phq8+1+0000000003+0000000003.avro
-rw-r--r--   3 root supergroup       4445 2018-03-22 09:32 /topicAndroidNew/questionnaire_phq8/partition=1/questionnaire_phq8+1+0000000004+0000000004.avro
-rw-r--r--   3 root supergroup       4445 2018-03-22 11:45 /topicAndroidNew/questionnaire_phq8/partition=1/questionnaire_phq8+1+0000000005+0000000005.avro
-rw-r--r--   3 root supergroup       4445 2018-03-22 13:35 /topicAndroidNew/questionnaire_phq8/partition=1/questionnaire_phq8+1+0000000006+0000000006.avro
-rw-r--r--   3 root supergroup       4445 2018-03-23 09:01 /topicAndroidNew/questionnaire_phq8/partition=1/questionnaire_phq8+1+0000000007+0000000007.avro
-rw-r--r--   3 root supergroup       4445 2018-03-26 08:37 /topicAndroidNew/questionnaire_phq8/partition=1/questionnaire_phq8+1+0000000008+0000000008.avro
-rw-r--r--   3 root supergroup       4390 2018-03-26 09:01 /topicAndroidNew/questionnaire_phq8/partition=1/questionnaire_phq8+1+0000000009+0000000009.avro
-rw-r--r--   3 root supergroup       4387 2018-03-26 10:51 /topicAndroidNew/questionnaire_phq8/partition=1/questionnaire_phq8+1+0000000010+0000000010.avro
-rw-r--r--   3 root supergroup       4445 2018-03-29 09:31 /topicAndroidNew/questionnaire_phq8/partition=1/questionnaire_phq8+1+0000000011+0000000011.avro
-rw-r--r--   3 root supergroup       4462 2018-03-30 09:53 /topicAndroidNew/questionnaire_phq8/partition=1/questionnaire_phq8+1+0000000012+0000000012.avro
-rw-r--r--   3 root supergroup       4462 2018-04-04 08:06 /topicAndroidNew/questionnaire_phq8/partition=1/questionnaire_phq8+1+0000000013+0000000013.avro
-rw-r--r--   3 root supergroup       4445 2018-04-05 09:01 /topicAndroidNew/questionnaire_phq8/partition=1/questionnaire_phq8+1+0000000014+0000000014.avro
-rw-r--r--   3 root supergroup       4710 2018-04-05 09:16 /topicAndroidNew/questionnaire_phq8/partition=1/questionnaire_phq8+1+0000000015+0000000016.avro
-rw-r--r--   3 root supergroup       5107 2018-04-09 07:56 /topicAndroidNew/questionnaire_phq8/partition=1/questionnaire_phq8+1+0000000017+0000000017.avro
-rw-r--r--   3 root supergroup       4462 2018-04-09 11:07 /topicAndroidNew/questionnaire_phq8/partition=1/questionnaire_phq8+1+0000000018+0000000018.avro
-rw-r--r--   3 root supergroup       4462 2018-04-11 18:47 /topicAndroidNew/questionnaire_phq8/partition=1/questionnaire_phq8+1+0000000019+0000000019.avro
-rw-r--r--   3 root supergroup       4462 2018-04-12 12:34 /topicAndroidNew/questionnaire_phq8/partition=1/questionnaire_phq8+1+0000000020+0000000020.avro
-rw-r--r--   3 root supergroup       4462 2018-04-17 19:16 /topicAndroidNew/questionnaire_phq8/partition=1/questionnaire_phq8+1+0000000021+0000000021.avro
-rw-r--r--   3 root supergroup       4462 2018-04-18 07:03 /topicAndroidNew/questionnaire_phq8/partition=1/questionnaire_phq8+1+0000000022+0000000022.avro

hdfs dfs -ls /topicAndroidNew/questionnaire_phq8/partition=1_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-1

-rw-r--r--   3 root supergroup       4972 2018-05-15 15:02 /topicAndroidNew/questionnaire_phq8/partition=1_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-1/questionnaire_phq8+1+0000000001+0000000003.avro
-rw-r--r--   3 root supergroup       6460 2018-05-16 11:00 /topicAndroidNew/questionnaire_phq8/partition=1_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-1/questionnaire_phq8+1+0000000001+0000000008.avro
-rw-r--r--   3 root supergroup       6987 2018-05-21 15:55 /topicAndroidNew/questionnaire_phq8/partition=1_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-1/questionnaire_phq8+1+0000000001+0000000010.avro
-rw-r--r--   3 root supergroup       4975 2018-05-15 22:17 /topicAndroidNew/questionnaire_phq8/partition=1_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-1/questionnaire_phq8+1+0000000004+0000000006.avro
-rw-r--r--   3 root supergroup       6212 2018-05-25 20:47 /topicAndroidNew/questionnaire_phq8/partition=1_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-1/questionnaire_phq8+1+0000000004+0000000010.avro
-rw-r--r--   3 root supergroup       6474 2018-05-29 15:28 /topicAndroidNew/questionnaire_phq8/partition=1_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-1/questionnaire_phq8+1+0000000004+0000000011.avro
-rw-r--r--   3 root supergroup       6902 2018-05-31 11:12 /topicAndroidNew/questionnaire_phq8/partition=1_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-1/questionnaire_phq8+1+0000000004+0000000012.avro
-rw-r--r--   3 root supergroup       7362 2018-06-03 15:45 /topicAndroidNew/questionnaire_phq8/partition=1_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-1/questionnaire_phq8+1+0000000004+0000000013.avro
-rw-r--r--   3 root supergroup       7635 2018-06-04 10:34 /topicAndroidNew/questionnaire_phq8/partition=1_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-1/questionnaire_phq8+1+0000000004+0000000014.avro
-rw-r--r--   3 root supergroup       4642 2018-05-15 22:32 /topicAndroidNew/questionnaire_phq8/partition=1_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-1/questionnaire_phq8+1+0000000007+0000000007.avro
-rw-r--r--   3 root supergroup       4462 2018-05-21 08:16 /topicAndroidNew/questionnaire_phq8/partition=1_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-1/questionnaire_phq8+1+0000000009+0000000009.avro
-rw-r--r--   3 root supergroup       6147 2018-06-20 10:46 /topicAndroidNew/questionnaire_phq8/partition=1_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-1/questionnaire_phq8+1+0000000009+0000000014.avro
-rw-r--r--   3 root supergroup       4459 2018-05-21 09:27 /topicAndroidNew/questionnaire_phq8/partition=1_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-1/questionnaire_phq8+1+0000000010+0000000010.avro
-rw-r--r--   3 root supergroup       4459 2018-05-28 09:37 /topicAndroidNew/questionnaire_phq8/partition=1_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-1/questionnaire_phq8+1+0000000011+0000000011.avro
-rw-r--r--   3 root supergroup       5620 2018-06-25 09:35 /topicAndroidNew/questionnaire_phq8/partition=1_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-1/questionnaire_phq8+1+0000000011+0000000014.avro
-rw-r--r--   3 root supergroup       4657 2018-06-02 18:25 /topicAndroidNew/questionnaire_phq8/partition=1_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-1/questionnaire_phq8+1+0000000013+0000000013.avro
-rw-r--r--   3 root supergroup       4470 2018-06-04 08:15 /topicAndroidNew/questionnaire_phq8/partition=1_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-1/questionnaire_phq8+1+0000000014+0000000014.avro

hdfs dfs -ls /topicAndroidNew/questionnaire_phq8/partition=1_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-2

Found 1 items
-rw-r--r--   3 root supergroup       5806 2018-06-20 11:01 /topicAndroidNew/questionnaire_phq8/partition=1_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-2/questionnaire_phq8+1+0000000015+0000000018.avro

hdfs dfs -ls /topicAndroidNew/questionnaire_phq8/partition=1_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-3

Found 1 items
-rw-r--r--   3 root supergroup       5806 2018-06-20 13:10 /topicAndroidNew/questionnaire_phq8/partition=1_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-3/questionnaire_phq8+1+0000000015+0000000018.avro

hdfs dfs -ls /topicAndroidNew/questionnaire_phq8/partition=1_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-4

Found 3 items
-rw-r--r--   3 root supergroup       6143 2018-06-25 09:50 /topicAndroidNew/questionnaire_phq8/partition=1_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-4/questionnaire_phq8+1+0000000015+0000000019.avro
-rw-r--r--   3 root supergroup       4797 2018-06-25 16:19 /topicAndroidNew/questionnaire_phq8/partition=1_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-4/questionnaire_phq8+1+0000000020+0000000020.avro
-rw-r--r--   3 root supergroup       4798 2018-06-26 21:40 /topicAndroidNew/questionnaire_phq8/partition=1_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-4/questionnaire_phq8+1+0000000021+0000000021.avro

hdfs dfs -ls /topicAndroidNew/questionnaire_phq8/partition=1_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-6

Found 1 items
-rw-r--r--   3 root supergroup       4969 2018-07-02 12:14 /topicAndroidNew/questionnaire_phq8/partition=1_key_schema=org.radarcns.kafka.ObservationKey-1_value_schema=org.radarcns.active.questionnaire.Questionnaire-6/questionnaire_phq8+1+0000000023+0000000023.avro

I know these folders are because of the custom partitioner which uses the schema version. The current latest schema version for phq8-value is 6. As you can see that in the above files for version 6, the offset is 23, so yes it get the correct next offset 24, but then it says Fetch offset 24 is out of range and then resets it to 11. Where is it getting this offset 11 and why does it reset it to 11 ?? This causes records from offset 11 to 23 to be re-processed.
@yatharthranjan
Copy link
Member Author

Could be because of tmp files as stated here -

https://github.com/confluentinc/kafka-connect-hdfs/blob/18e5149859867fdb1e6002ad063be27dd91f5180/src/main/java/io/confluent/connect/hdfs/TopicPartitionWriter.java#L614

@blootsvoets
Copy link
Member

blootsvoets commented Jul 4, 2018 •
edited
Loading

This is due to confluentinc/kafka-connect-hdfs#224. I'm looking into the code whether there is a fix.

@yatharthranjan
Copy link
Member Author

But in our case, i think this is not the issue since the url/topic.dir/topic.name is same even with the custom partitioner. Just the name of the partition is changed by the custom partitioner.
Looking at the code, ideally the connector should traverse all the files under the url/topic.dir/topic.name directory no matter which sub-directory and fetch the max offset.
Does that make sense ?

Take a look at this function which fetches the max offset - https://github.com/confluentinc/kafka-connect-hdfs/blob/18e5149859867fdb1e6002ad063be27dd91f5180/src/main/java/io/confluent/connect/hdfs/FileUtils.java#L106

@blootsvoets
Copy link
Member

blootsvoets commented Jul 4, 2018 •
edited
Loading

I don't think it is. The TopicPartitionWriter will use the partition path, and that is

public String generatePartitionedPath(String topic, String encodedPartition) {
      return topic + "/" + encodedPartition;
}

which in turn uses our Schema-based partioning. I've done a test and it seems like our partitioner is no longer needed. The bug we needed it for is fixed upstream. I'm creating a integration test as well to show this.

@blootsvoets blootsvoets mentioned this issue Jul 4, 2018
Merged
@yatharthranjan
Copy link
Member Author

This was fixed by using the default partitioner as in #21 and deleting old stale temp files.
Will also need to delete the schema based partitions from HDFS after the normal partitions are up to date.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@blootsvoets @yatharthranjan