KIP 714 with compression support #4721

anchitj · 2024-05-17T06:31:06Z

No description provided.

…data Response.

* WIP:Push telemetry is being scheduled now. * Push * Compliation Fix * Working * Use UUID in PUSH * Remove fprintf * Changes * Fix CONFIGURATION.md * Fix size * Update s2i bounds --------- Co-authored-by: Milind L <miluthra@confluent.io>

* Add broker selection and client termination * Address review comments

* Serialise metrics using nanopb * Move nanopb and opentelemetry inside src * Add metrics.options file * Remove unused includes * Style fix * Fix formatting * Skip copyright check * Add nanopb and opentelemetry in windows vcxproj * Include headers directories in CMAKE * Use flexver with PushTelemetry * Fix memory leaks * Change import path * Use rd_bool_t everywhere * Fix librdkafka.vcxproj * Use rd_bool_t * PR Feedback * Add nanopb license * Include opentelemtry license

* Support for delta temporality * Style fix * Fix bugs * Fix memory leaks and formatting * Fixes * PR Feedback

* Add telemetry encode and decode unit tests * Style fix * Improve test * PR Feedback

* Add max telemetry bytes * Clear telemetry_max_bytes * PR comments

…master_merge_with_compression

emasab

Comments about encode and decode

src/rdkafka_op.h

src/rdkafka_telemetry_encode.c

src/rdkafka_telemetry_decode.h

src/rdkafka_telemetry_encode.c

emasab

Most of remaining comments

src/rdkafka_broker.c

src/rdkafka_int.h

src/rdkafka_mock_handlers.c

src/rdkafka_telemetry.c

emasab

Comments about unit tests and naming

src/rdkafka_telemetry_decode.c

src/rdkafka_telemetry.c

src/rdkafka_telemetry_encode.c

src/rdkafka_mock_handlers.c

…master_merge_with_compression

emasab

Copyright and includes comments

src/rd.h

src/rdkafka_telemetry_decode.c

src/rdkafka_telemetry_decode.h

emasab · 2024-07-05T08:54:40Z

src/rdkafka_telemetry_encode.h

+#ifndef _RDKAFKA_RDKAFKA_TELEMETRY_ENCODE_H
+#define _RDKAFKA_RDKAFKA_TELEMETRY_ENCODE_H
+
+#include "rdtypes.h"


Add #include "rdkafka_int.h" as it's needed for rd_kafka_t

Suggested change

#include "rdtypes.h"

#include "rdkafka_int.h"

#include "rdtypes.h"

src/rdkafka_telemetry_encode.c

src/rdkafka_telemetry.c

emasab · 2024-07-05T08:58:27Z

src/rdkafka_telemetry.h

+
+
+#ifndef _RD_KAFKA_TELEMETRY_H_
+#define _RD_KAFKA_TELEMETRY_H_


Required for rd_kafka_t

Suggested change

#define _RD_KAFKA_TELEMETRY_H_

#define _RD_KAFKA_TELEMETRY_H_

#include "rdkafka_int.h"

src/rdkafka_telemetry.h

src/rdkafka_telemetry_decode.c

emasab · 2024-07-07T15:22:11Z

Please add these to the list of Supported protocol versions in INTRODUCTION.md

| 71      | GetTelemetrySubscriptions     | 0          | 0              |
| 72      | PushTelemetry                 | 0          | 0              |

emasab

Some comments about placeholders types in formatted strings

src/rdkafka_telemetry_decode.c

src/rdkafka_telemetry_encode.c

src/rdkafka_telemetry.c

emasab

Awesome work, thanks @anchitj and @milindl !

georgthegreat · 2024-07-22T15:24:40Z

Hmm...
This PR introduced a mandatory dependency on nanopb runtime and opentelemetry protos generated with nanopb.
While there is nothing wrong with either of dependencies, having a switch to disable them would be nice.

@anchitj @emasab what do you think?
Other depenndencies are configurable.

milindl and others added 30 commits July 25, 2023 12:06

Add boilerplate for the entire flow (#4365)

3f0eed3

Added new data type UUID for topic id. Added topic id support in Meta…

56fa55a

…data Response.

Merge Get and Push Protocol (#4377)

9546d71

* WIP:Push telemetry is being scheduled now. * Push * Compliation Fix * Working * Use UUID in PUSH * Remove fprintf * Changes * Fix CONFIGURATION.md * Fix size * Update s2i bounds --------- Co-authored-by: Milind L <miluthra@confluent.io>

Clang format fixes

e11aa51

Fix memory leaks (#4378)

ca6a8e5

Add broker selection and client termination [KIP-714] (#4382)

ef07a95

* Add broker selection and client termination * Address review comments

Handle delta temporality and some code refactoring (#4410)

5c943ba

* Support for delta temporality * Style fix * Fix bugs * Fix memory leaks and formatting * Fixes * PR Feedback

Use rd_kafka_compression_t (#4419)

9e3cf26

Add method to get requests from mock cluster

21f7072

Fix to ns

f1c4fc4

Add capacity in mock broker to change PushIntervalMs

7fdbfbe

Fix memory leak

e6a9d66

Fix jitter: should not be decimal

e0b4f3e

Add test for telemetry using mock broker

d7e4b47

Address comments

a614d2c

Unit tests push telemetry encode decode (#4440)

1216b39

* Add telemetry encode and decode unit tests * Style fix * Improve test * PR Feedback

Initial code for metrics compression

4ed6abd

Style and some bug fixes

3d66f64

Little refactoring

88dcfc1

Add max telemetry bytes (#4507)

81913fe

* Add max telemetry bytes * Clear telemetry_max_bytes * PR comments

Add other producer metrics

ccc1a26

More changes

b6c04ac

Add enable.metrics.push config

94545fe

Fix bugs

aa8a668

Refactor

8c38f8e

Fix warnings

3987849

Fix UT

4d1d192

Nullify after free

2f6afa2

Fix bugs

eff9351

anchitj added 3 commits July 3, 2024 13:17

Move first_push to telemetry.c

387ae01

Merge branch 'master' into dev_kip_714_mock_broker_integration_tests_…

7a36b53

…master_merge_with_compression

Fix unit test

77a75b4

emasab requested changes Jul 3, 2024

View reviewed changes

anchitj added 2 commits July 3, 2024 16:35

PR feedback 5th round

23c3b8f

Rename interface to decode_interface

69f9dcf

emasab requested changes Jul 3, 2024

View reviewed changes

anchitj added 2 commits July 3, 2024 23:08

PR feedback 6th round

77ce834

PR feedback

1924e3f

emasab requested changes Jul 4, 2024

View reviewed changes

PR feedback 7th round

476038a

emasab reviewed Jul 5, 2024

View reviewed changes

src/rdkafka_mock_handlers.c Show resolved Hide resolved

anchitj added 3 commits July 5, 2024 12:28

PR feedback 8th round

dd7d1fc

Merge branch 'master' into dev_kip_714_mock_broker_integration_tests_…

bb53d5e

…master_merge_with_compression

Change failure message in rd_assert

b78751c

emasab requested changes Jul 5, 2024

View reviewed changes

anchitj added 5 commits July 5, 2024 15:04

PR feedback 9th round

652b0ea

Update copyright year rdkafka_conf.h

40b0ad8

Remove arg being cast as test_data

46e4579

Add changelog entry

0fbc703

Remove changes to 0009-mock_cluster.c

bbd316b

emasab requested changes Jul 7, 2024

View reviewed changes

PR feedback 10th round

4732ff6

anchitj force-pushed the dev_kip_714_mock_broker_integration_tests_master_merge_with_compression branch from 9616eba to 4732ff6 Compare July 8, 2024 06:44

emasab approved these changes Jul 8, 2024

View reviewed changes

anchitj merged commit 6eaf89f into master Jul 8, 2024
2 checks passed

anchitj deleted the dev_kip_714_mock_broker_integration_tests_master_merge_with_compression branch July 8, 2024 07:47

MarcAurel mentioned this pull request Jul 23, 2024

Unable to build without SSL since #4721 #4793

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KIP 714 with compression support #4721

KIP 714 with compression support #4721

anchitj commented May 17, 2024

emasab left a comment

emasab left a comment

emasab left a comment

emasab left a comment

emasab Jul 5, 2024

emasab Jul 5, 2024

emasab commented Jul 7, 2024

emasab left a comment

emasab left a comment

georgthegreat commented Jul 22, 2024

	#include "rdtypes.h"
	#include "rdkafka_int.h"
	#include "rdtypes.h"



		#ifndef _RD_KAFKA_TELEMETRY_H_
		#define _RD_KAFKA_TELEMETRY_H_

KIP 714 with compression support #4721

KIP 714 with compression support #4721

Conversation

anchitj commented May 17, 2024

emasab left a comment

Choose a reason for hiding this comment

emasab left a comment

Choose a reason for hiding this comment

emasab left a comment

Choose a reason for hiding this comment

emasab left a comment

Choose a reason for hiding this comment

emasab Jul 5, 2024

Choose a reason for hiding this comment

emasab Jul 5, 2024

Choose a reason for hiding this comment

emasab commented Jul 7, 2024

emasab left a comment

Choose a reason for hiding this comment

emasab left a comment

Choose a reason for hiding this comment

georgthegreat commented Jul 22, 2024