Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix/william/genrpc ssl port align #11813

Merged
merged 2 commits into from Oct 25, 2023
Merged

Fix/william/genrpc ssl port align #11813

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2a7d73e
ci: fix helm chart test
qzhuyan Oct 24, 2023
6346e0d
fix(gen_rpc): ssl client port align with server port
qzhuyan Oct 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/run_helm_tests.yaml
View file
Open in desktop
Expand Up @@ -89,6 +89,7 @@ jobs:
EMQX_RPC__CACERTFILE: /opt/emqx/etc/certs/cacert.pem
EMQX_RPC__CIPHERS: TLS_AES_256_GCM_SHA384,TLS_AES_128_GCM_SHA256
EMQX_RPC__TLS_VERSIONS: "[tlsv1.2]"
EMQX_RPC__PORT_DISCOVERY: manual
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this for redundance ? because it's set to manual in docker entrypoint script.
https://github.com/emqx/emqx/blob/master/deploy/docker/docker-entrypoint.sh#L52

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I thought it was auto-discovery as it is the default in the schema. I set it to manual want to test both. shall we remove it from docker-entrypoint? I don't see good reason to have it there.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or we should change the schema default to manual

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The override in docker entrypoint was added long ago, we keep it for backward compatibility reasons. (so that v4 and v5 has the same port mapping convention).

We cannot change the default in schema because for RPM/DEB/TGZ installation, it's offset based port mapping. --- also kept for backward compatibility reasons.

EOL
- name: run emqx on chart (k8s)
if: matrix.discovery == 'k8s'
Expand Down
4 changes: 4 additions & 0 deletions apps/emqx_conf/src/emqx_conf_schema.erl
View file
Open in desktop
Expand Up @@ -1151,6 +1151,7 @@ translation("emqx") ->
translation("gen_rpc") ->
[
{"default_client_driver", fun tr_gen_rpc_default_client_driver/1},
{"ssl_client_port", fun tr_gen_rpc_ssl_client_port/1},
Copy link
Member

@zmstone zmstone Oct 25, 2023 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is probably the only change needed.

or fix this instead: gen_rpc has a default value for ssl clients to connect to 5370
https://github.com/emqx/gen_rpc/blob/46dd22100391f3d87daca989cde7e2ecc52a17b5/src/gen_rpc.app.src#L35

but we have set the same server port on 5369 for both tcp and ssl:

emqx/apps/emqx_conf/src/emqx_conf_schema.erl

Lines 798 to 813 in a4ab744

{"tcp_server_port",
sc(
integer(),
#{
mapping => "gen_rpc.tcp_server_port",
default => 5369,
desc => ?DESC(rpc_tcp_server_port)
}
)},
{"ssl_server_port",
sc(
integer(),
#{
mapping => "gen_rpc.ssl_server_port",
default => 5369,
desc => ?DESC(rpc_ssl_server_port)

Copy link
Member

@zmstone zmstone Oct 25, 2023 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for completeness, we should actually add tcp_client_port too
otherwise manual will not work if one wants to change port for server.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually, to support manual strategy properly, it should not be a simple translation and user server port.
should rather make the tcp and ssl ports configurable for the clients.

Copy link
Member

@zmstone zmstone Oct 25, 2023 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we force all nodes in the cluster to use the same server port, then maybe just a translation is enough.

but please also add the same tr for tcp_client_port.

{"ssl_client_options", fun tr_gen_rpc_ssl_options/1},
{"ssl_server_options", fun tr_gen_rpc_ssl_options/1},
{"socket_ip", fun(Conf) ->
Expand Down Expand Up @@ -1226,6 +1227,9 @@ collector_enabled(disabled, _) -> [].
tr_gen_rpc_default_client_driver(Conf) ->
conf_get("rpc.protocol", Conf).

tr_gen_rpc_ssl_client_port(Conf) ->
conf_get("rpc.ssl_server_port", Conf).
zmstone marked this conversation as resolved.
Show resolved Hide resolved

tr_gen_rpc_ssl_options(Conf) ->
Ciphers = conf_get("rpc.ciphers", Conf),
Versions = conf_get("rpc.tls_versions", Conf),
Expand Down
4 changes: 4 additions & 0 deletions changes/ce/fix-11813.en.md
View file
Open in desktop
@@ -0,0 +1,4 @@
Fix schema: RPC client ssl port alighn with configured server port.
And ensure RPC ports are opened in helm chart.


4 changes: 4 additions & 0 deletions deploy/charts/emqx-enterprise/templates/StatefulSet.yaml
View file
Open in desktop
Expand Up @@ -121,6 +121,10 @@ spec:
{{- end }}
- name: ekka
containerPort: 4370
- name: genrpc-manual
containerPort: 5369
- name: genrpc2-auto
containerPort: 5370
Comment on lines +126 to +127
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there should be no need for this. ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need it for port port_discovery = stateless in gen_rpc, yes this is confusion.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/emqx/gen_rpc/blob/46dd22100391f3d87daca989cde7e2ecc52a17b5/README.md?plain=1#L117

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

with the override set in docker entrypoint, i don't think we need this port.

envFrom:
- configMapRef:
name: {{ include "emqx.fullname" . }}-env
Expand Down
4 changes: 4 additions & 0 deletions deploy/charts/emqx/templates/StatefulSet.yaml
View file
Open in desktop
Expand Up @@ -121,6 +121,10 @@ spec:
{{- end }}
- name: ekka
containerPort: 4370
- name: genrpc-manual
containerPort: 5369
- name: genrpc2-auto
containerPort: 5370
Comment on lines +126 to +127
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here

envFrom:
- configMapRef:
name: {{ include "emqx.fullname" . }}-env
Expand Down