feat: add connector schema scaffold and break out Kafka conector #11581
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kjellwinblad
force-pushed
the
kjell/break_out_connection_settings_from_bridge/EMQX-10805/EMQX-10770
branch
from
5c5baac
to
05c4348
Compare
kjellwinblad
force-pushed
the
kjell/break_out_connection_settings_from_bridge/EMQX-10805/EMQX-10770
branch
from
765b2d3
to
0953a8a
Compare
zmstone
reviewed
Oct 11, 2023
zmstone
reviewed
Oct 11, 2023
Comment on lines
+515
to
+553
| false -> | ||
| {error, | ||
| <<<<"on_add_channel callback function not available for connector with resource id ">>/binary, | ||
| ResId/binary>>} |
There was a problem hiding this comment.
if this happens, isn't it a bug?
we should perhaps crash it or perform higher level sanity check?
There was a problem hiding this comment.
I agree, we could crash here. Will fix
zmstone
reviewed
Oct 11, 2023
Comment on lines
66
to
67
| %% TODO: rename this to `kafka_producer' after alias support is added | ||
| %% to hocon; keeping this as just `kafka' for backwards compatibility. |
There was a problem hiding this comment.
now it's the time for this TODO?
There was a problem hiding this comment.
Yes, this is probably a good time. We can try to do this before the merge.
kjellwinblad
force-pushed
the
kjell/break_out_connection_settings_from_bridge/EMQX-10805/EMQX-10770
branch
3 times, most recently
from
7fb736e
to
4f04afb
Compare
kjellwinblad
force-pushed
the
kjell/break_out_connection_settings_from_bridge/EMQX-10805/EMQX-10770
branch
from
b88f282
to
e3260a5
Compare
HJianBo
reviewed
Oct 18, 2023
kjellwinblad
force-pushed
the
kjell/break_out_connection_settings_from_bridge/EMQX-10805/EMQX-10770
branch
from
f843b52
to
a2cb00b
Compare
id
force-pushed
the
kjell/break_out_connection_settings_from_bridge/EMQX-10805/EMQX-10770
branch
from
a70de06
to
26f0cb8
Compare
thalesmg
reviewed
Oct 25, 2023
zmstone
force-pushed
the
kjell/break_out_connection_settings_from_bridge/EMQX-10805/EMQX-10770
branch
from
32f01cd
to
1ce7c2e
Compare
This commit is the beginning of an effort to split bridges into a connector part and a bridge part. Several bridges should be able to share a connector pool defined by a single connector. The connectors should be possible to enable and disable similar to how one can disable and enable bridges. There should also be an API for checking the status of a connector and for add/edit/delete connectors similar to the current bridge API. Issues: https://emqx.atlassian.net/browse/EMQX-10805
* Most Bridge V1 HTTP API calls are now compatible with Bridge V2 * Local topics works for Bridge V2 now * A lot of work on trying to get the old Kafka producer test suite to work after the refactorings
Prior to this fix, Kafka producer config was used as client config
zmstone
force-pushed
the
kjell/break_out_connection_settings_from_bridge/EMQX-10805/EMQX-10770
branch
from
85d0876
to
5f17a8f
Compare
Most operation for the bridge V1 HTTP API compatibility layer are now working. This has been tested by creating/deleting/updating Kafka bridge through HTTP API, sending message to it, and resetting and checking metrics. The start, stop, restart, enable, disable operations still need to be fixed.
* Make sure that a rule that refer to a bridge that has been converted to a bridge V2 bridge gets its type converted if needed. * Add test case for sending message to a Bridge V2 through a rule
zmstone
approved these changes
Oct 30, 2023
|
Closing this since @id has squashed and merged to release-53 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This work in progress draft PR is the beginning of an effort to split bridges into a connector part and a bridge part.
Several bridges should be able to share a connector pool defined by a single connector. The connectors should be possible to enable and disable similar to how one can disable and enable bridges. There should also be an API for checking the status of a connector and for add/edit/delete connectors similar to the current bridge API.
Fixes:
https://emqx.atlassian.net/browse/EMQX-10953
NOTE:
This PR still requires a lot of work and is just created to increase transparency while doing the work.
Summary
🤖 Generated by Copilot at 02cd5d0
This pull request adds hocon schema support for the kafka connector configuration in the enterprise edition of emqx. It refactors and documents the connector_config fields in
emqx_bridge_kafka.erland adds new schema modules inemqx_confandemqx_connectorapplications.PR Checklist
Please convert it to a draft if any of the following conditions are not met. Reviewers may skip over until all the items are checked:
changes/(ce|ee)/(feat|perf|fix)-<PR-id>.en.mdfilesChecklist for CI (.github/workflows) changes
changes/dir for user-facing artifacts update