[FLINK-24409][kafka] Fix collection of KafkaSourceReaderMetrics for t… #17773

fapaul · 2021-11-12T08:31:10Z

…opics containing periods

What is the purpose of the change

Internally, Kafka translates the periods in topic names to underscore.
This led to that Flink could not collect the metrics and logged a
warning. With this commit, we also translate the topic name before trying
to collect the metrics.

This PR subsumes #17401

Brief change log

Translate the periods in topic names to underscore for metric collection

Verifying this change

Added a test to verify the log collection of a topic containing periods

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (yes / no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
The serializers: (yes / no / don't know)
The runtime per-record code paths (performance sensitive): (yes / no / don't know)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
The S3 file system connector: (yes / no / don't know)

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

flinkbot · 2021-11-12T08:34:22Z

CI report:

f6c57af Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

flinkbot · 2021-11-12T08:36:08Z

Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.

Automated Checks

Last check on commit e53ad5e (Fri Nov 12 08:36:08 UTC 2021)

Warnings:

No documentation files were touched! Remember to keep the Flink docs up to date!

_{Mention the bot in a comment to re-run the automated checks.}

Review Progress

❓ 1. The [description] looks good.
❓ 2. There is [consensus] that the contribution should go into to Flink.
❓ 3. Needs [attention] from.
❓ 4. The change fits into the overall [architecture].
❓ 5. Overall code [quality] is good.

Please see the Pull Request Review Guide for a full explanation of the review process.

The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot approve description to approve one or more aspects (aspects: description, consensus, architecture and quality)
@flinkbot approve all to approve all aspects
@flinkbot approve-until architecture to approve everything until architecture
@flinkbot attention @username1 [@username2 ..] to require somebody's attention
@flinkbot disapprove architecture to remove an approval you gave earlier

JingGe · 2021-11-15T09:31:42Z

.../src/main/java/org/apache/flink/connector/kafka/source/metrics/KafkaSourceReaderMetrics.java

                                && tags.containsKey("partition")
-                                && tags.get("partition").equals(String.valueOf(tp.partition()));
+                                && tags.get("partition").equals(resolvedPartition);


May I know the reason of introducing local variables instead of change it directly?

I do not think there is any good reason besides keeping the line a bit shorter.

JingGe · 2021-11-15T09:53:14Z

.../test/java/org/apache/flink/connector/kafka/source/reader/KafkaPartitionSplitReaderTest.java

        // Validate pendingRecords
        for (int i = 0; i < NUM_RECORDS_PER_PARTITION; i++) {
            reader.fetch();
-            assertEquals(NUM_RECORDS_PER_PARTITION - i - 1, (long) pendingRecords.get().getValue());
+            Assertions.assertEquals(
+                    NUM_RECORDS_PER_PARTITION - i - 1, (long) pendingRecords.get().getValue());


Nit: all the assert changes could be avoided by using static import. This will help reducing the review effort and minimise any unexpected human errors, e.g. typo.

JingGe · 2021-11-15T09:55:44Z

.../test/java/org/apache/flink/connector/kafka/source/reader/KafkaPartitionSplitReaderTest.java

+                    topic1Name, true, true, KafkaSourceTestEnv::getRecordsForTopic);
+            KafkaSourceTestEnv.setupTopic(
+                    topic2Name, true, true, KafkaSourceTestEnv::getRecordsForTopic);
+        }


Does it mean that the rest of the test works when topicSuffix.isEmpty() == true for @EmptySource and no topic has been created?

Yes the test essentially setups two additional topics if the suffix is specified. TOPIC1 and TOPIC2 are already set up in the setup method.

got it, thanks!

fapaul

Thanks for the review. PTAL

fapaul · 2021-11-15T10:02:54Z

.../test/java/org/apache/flink/connector/kafka/source/reader/KafkaPartitionSplitReaderTest.java

+                    topic1Name, true, true, KafkaSourceTestEnv::getRecordsForTopic);
+            KafkaSourceTestEnv.setupTopic(
+                    topic2Name, true, true, KafkaSourceTestEnv::getRecordsForTopic);
+        }


Yes the test essentially setups two additional topics if the suffix is specified. TOPIC1 and TOPIC2 are already set up in the setup method.

fapaul · 2021-11-15T10:12:16Z

.../src/main/java/org/apache/flink/connector/kafka/source/metrics/KafkaSourceReaderMetrics.java

                                && tags.containsKey("partition")
-                                && tags.get("partition").equals(String.valueOf(tp.partition()));
+                                && tags.get("partition").equals(resolvedPartition);


I do not think there is any good reason besides keeping the line a bit shorter.

JingGe · 2021-11-15T10:56:05Z

Thanks for your contribution, LGTM.

mas-chen

lgtm, is it also possible to fix https://github.com/apache/flink/pull/17773/files#diff-1b14f8cbd74aef6cdc991d1bef005bf0208d0a0a43d33807e13808550e82adc0R313 which is the log message is shown in the original ticket.

I believe it is MetricNames.PENDING_RECORDS that would be reported incorrectly.

fapaul · 2021-11-16T08:17:38Z

lgtm, is it also possible to fix https://github.com/apache/flink/pull/17773/files#diff-1b14f8cbd74aef6cdc991d1bef005bf0208d0a0a43d33807e13808550e82adc0R313 which is the log message is shown in the original ticket.

I believe it is MetricNames.PENDING_RECORDS that would be reported incorrectly.

@mas-chen Thanks for the reminder but I am not sure which log message you are referring to. Unfortunately, the link you posted does not point to a line.

mas-chen · 2021-11-16T18:58:52Z

@fapaul this is the one I am referring to:

flink/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/metrics/KafkaSourceReaderMetrics.java

Line 313 in d2e4359

RECORDS_LAG, tp, MetricNames.PENDING_BYTES),

…opics containing periods Internally, Kafka translates the periods in topic names to underscore. This led to that Flink could not collect the metrics and logged a warning. With this commit we also translate the topic name before trying to collect the metrics.

…llection fails

rmetzger added the component=Connectors/Kafka label Nov 12, 2021

JingGe reviewed Nov 15, 2021

View reviewed changes

fapaul force-pushed the FLINK-24409 branch from e53ad5e to d2e4359 Compare November 15, 2021 10:11

fapaul commented Nov 15, 2021

View reviewed changes

fapaul mentioned this pull request Nov 15, 2021

[FLINK-24409][kafka] Fix collection of KafkaSourceReaderMetrics for topics containing periods #17799

Merged

mas-chen reviewed Nov 15, 2021

View reviewed changes

Fabian Paul added 2 commits November 17, 2021 10:57

[FLINK-24409][kafka] Log PendingRecords metrics name if record lag co…

f6c57af

…llection fails

fapaul force-pushed the FLINK-24409 branch from d2e4359 to f6c57af Compare November 17, 2021 09:57

mas-chen approved these changes Nov 17, 2021

View reviewed changes

AHeise merged commit da8ef26 into apache:master Nov 17, 2021

jherico mentioned this pull request Nov 17, 2021

[FLINK-24409][connectors] Fix metrics errors with topics names with periods #17401

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLINK-24409][kafka] Fix collection of KafkaSourceReaderMetrics for t… #17773

[FLINK-24409][kafka] Fix collection of KafkaSourceReaderMetrics for t… #17773

fapaul commented Nov 12, 2021

flinkbot commented Nov 12, 2021 •

edited

flinkbot commented Nov 12, 2021

JingGe Nov 15, 2021

fapaul Nov 15, 2021

JingGe Nov 15, 2021

JingGe Nov 15, 2021

fapaul Nov 15, 2021

JingGe Nov 15, 2021

fapaul left a comment

fapaul Nov 15, 2021

fapaul Nov 15, 2021

JingGe commented Nov 15, 2021

mas-chen left a comment

fapaul commented Nov 16, 2021 •

edited

mas-chen commented Nov 16, 2021

[FLINK-24409][kafka] Fix collection of KafkaSourceReaderMetrics for t… #17773

[FLINK-24409][kafka] Fix collection of KafkaSourceReaderMetrics for t… #17773

Conversation

fapaul commented Nov 12, 2021

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

flinkbot commented Nov 12, 2021 • edited

CI report:

flinkbot commented Nov 12, 2021

Automated Checks

Review Progress

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fapaul left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JingGe commented Nov 15, 2021

mas-chen left a comment

Choose a reason for hiding this comment

fapaul commented Nov 16, 2021 • edited

mas-chen commented Nov 16, 2021

flinkbot commented Nov 12, 2021 •

edited

fapaul commented Nov 16, 2021 •

edited