Disable internal authentification (kerberos) between corrdinator and workers

[Consultante-yr]

Hello,

We have actually Prestodb with this version 0.215, and the internal authentification is configured with kerberos.
We want to disable this authentification because because there are too many Kerberos requests which are beating up the server.
So when i tried to delete this part related to kerberos in config.properties in coordinator and workers:
http-server.authentication.type=PASSWORD,KERBEROS
http.server.authentication.krb5.service-name=xxxxx
http.server.authentication.krb5.keytab=xxxx.keytab
http.authentication.krb5.config=/etc/krb5.conf
internal-communication.kerberos.enabled=true
I no longer have access to the ui and I have 503.
So what's the possibility to do disable the service without impacting the UI Please?
I saw in this link https://prestodb.io/docs/current/security/internal-communication.html#jwt, that i can use JWT but it's not compatible with the version we have.
I saw that i can replace http-server.authentication.type=PASSWORD,KERBEROS with http-erver.authentication.type=PASSWORD or http-erver.authentication.type=PASSWORD, CERTIFICAT but i get this https://coordiantor/v1/info/state returned status 401: Unauthorized"

For information: we use ldaps and tls/ssl in the configuration
Please any idea?

Thank you in advance,

[yhwang]

Hi @Consultante-yr, I assume you still need kerberos in your coordinator which protects your presto cluster. To disable the internal authentication with kerberos, you only need to remove internal-communication.kerberos.enabled on your coordinator and works. And for workers, since they talk to the coordinator over unauthenticated HTTP, you don't need the following kerberos settings:

• http-server.authentication.type
• http.server.authentication.krb5.service-name
• http.server.authentication.krb5.keytab
• http.authentication.krb5.config

[Consultante-yr]

Hello @yhwang,

Thank you for your response.
But when i removed internal-communication.kerberos.enabled on the coordinator, i get 0 active workers and in the log i encounter this error:
"WARN http-client-node-manager-56 com.facebook.presto.metadata.RemoteNodeState Error fetching node state from https://coordinator.domaine.com:8443/v1/info/state returned status 401: Unauthorized"

This is my file config in coordinator:
--- Im a coordinator
coordinator=true
discovery-server.enabled=true
node-scheduler.include-coordinator=false

---- discovery server
discovery.uri=https://coordinator.domaine.com:8443

http-server.authentication.type=PASSWORD,KERBEROS
http.server.authentication.krb5.service-name=prestodb
http.server.authentication.krb5.keytab=/etc/prestodb.keytab
http.authentication.krb5.config=/etc/krb5.conf
internal-communication.kerberos.enabled=true

---- ssl/tls

http-server.https.enabled=true
http-server.https.port=8443
http-server.https.keystore.path=/etc/prestodb.jks
http-server.https.keystore.key=xxxxxxx
http-server.http.enabled=false
internal-communication.https.required=true
internal-communication.https.keystore.path=/etc/prestodb.jks
internal-communication.https.keystore.key=xxxxx

---- config query
query.client.timeout=120.00m
query.max-memory=4GB
query.max-memory-per-node=4GB
query.max-total-memory=4GB
query.max-total-memory-per-node=4GB

node-scheduler.network-topology=flat

And when i tried also to remove internal-communication.kerberos.enabled in the workers,
I get this error:
" ERROR Announcer-0 io.airlift.discovery.client.Announcer Cannot connect to discovery server for announce: Announcement failed with status code 401:
2024-02-07T15:54:49.126Z ERROR Announcer-0 io.airlift.discovery.client.Announcer Service announcement failed after 51.13ms. Next request will happen within 0.00s"

This is my file config in Workers:

--- Im a worker
coordinator=false

--- discovery server
discovery.uri=https://coordinator.domaine.com:8443

http-server.authentication.type=PASSWORD,KERBEROS
http.server.authentication.krb5.service-name=prestodb
http.server.authentication.krb5.keytab=/etc/prestodb.keytab
http.authentication.krb5.config=/etc/krb5.conf
internal-communication.kerberos.enabled=false

---- ssl/tls

http-server.https.enabled=true
http-server.https.port=8443
http-server.https.keystore.path=/etc/prestodb.jks
http-server.https.keystore.key=keystore_password=xxxxxx
http-server.http.enabled=false
internal-communication.https.required=true
internal-communication.https.keystore.path=/etc/prestodb.jks
internal-communication.https.keystore.key=xxxx

---- config query
query.client.timeout=120.00m
query.max-memory=2GB
query.max-memory-per-node=1GB
query.max-total-memory=4GB
query.max-total-memory-per-node=1GB

node-scheduler.network-topology=flat
exchange.http-client.request-timeout=20s

And for workers, they communicate with the coordinator over HTTPS. I prefer to maintain HTTPS, as changing, it would require adjusting all configurations under load balancers.

Any Suggestions, please?

Thank you in advance,

Best Regards,

[yhwang]

The following suggestion is based on the assumption that you want to disable the Kerberos between the coordinator and workers but still enable Kerberos on the Presto UI:

In the coordinator:

coordinator=true
discovery-server.enabled=true
node-scheduler.include-coordinator=false

discovery.uri=https://coordinator.domaine.com:8443

Change discovery.uri to http://coordinator.domaine.com:8080 to use HTTP between the coordinator and workers. Pick a port number you can use on the coordinator and make sure all workers use the same discovery.uri

http-server.authentication.type=PASSWORD,KERBEROS
http.server.authentication.krb5.service-name=prestodb
http.server.authentication.krb5.keytab=/etc/prestodb.keytab
http.authentication.krb5.config=/etc/krb5.conf
internal-communication.kerberos.enabled=true

Change internal-communication.kerberos.enabled to false to disable the kerberos between the coordinator and workers. Or you can comment out this setting.

---- ssl/tls

http-server.https.enabled=true
http-server.https.port=8443
http-server.https.keystore.path=/etc/prestodb.jks
http-server.https.keystore.key=xxxxxxx
http-server.http.enabled=false

You still need HTTP server, so set this value to true: http-server.http.enabled=true
and assign a port number which matches the port number you use in the discovery.uri
For example:
http-server.http.port=8080

internal-communication.https.required=true

Set internal-communication.https.required to false and comment out the following two settings

#internal-communication.https.keystore.path=/etc/prestodb.jks
#internal-communication.https.keystore.key=xxxxx

Make sure the two settings above are commented out

====================================================

In the workers:

coordinator=false

--- discovery server
discovery.uri=https://coordinator.domaine.com:8443

Change to HTTP protocol and use the value from the coordinator. i.e: discovery.uri=http://coordinator.domaine.com:8080

Then comment out the following settings:

#http-server.authentication.type=PASSWORD,KERBEROS
#http.server.authentication.krb5.service-name=prestodb
#http.server.authentication.krb5.keytab=/etc/prestodb.keytab
#http.authentication.krb5.config=/etc/krb5.conf
#internal-communication.kerberos.enabled=false

#http-server.https.enabled=true
#http-server.https.port=8443
#http-server.https.keystore.path=/etc/prestodb.jks
#http-server.https.keystore.key=keystore_password=xxxxxx
#http-server.http.enabled=false
#internal-communication.https.required=true
#internal-communication.https.keystore.path=/etc/prestodb.jks
#internal-communication.https.keystore.key=xxxx

finally, you still need to set up http-server.http.port in the workers. i.e: http-server.http.port=8080 as long as the port number is available on the workers. Of course, each worker can use different port.

Again, I assume that you only want the Kerberos on the presto UI but no HTTPS and no Kerberos between the coordinator and workers.

[Consultante-yr]

Hello @yhwang,

Thank you for you answer.

I put the conf on the coordinator and the worker as you mentioned before and effectively the UI which works and the workers connected to it.
By launching curl I also see the number and names of the connected workers. However when I connect to the presto server cli and I launch a command like: "select * from system.runtime.nodes;", there is nothing is displayed either in the logs or on the terminal, and after a little while I have "Query is gone (server restarted?)", I don't see what this is linked to especially since I don't have any error that is displayed in particular. do you have an idea please?

otherwise if you want the logs I could also send them to you.

Thanks in advance,
Best Regards,

[yhwang]

@Consultante-yr sorry for the late response. The internal communication works properly on your Presto cluster. I suggest that follow this doc to enable the Kerberos for external communication of the coordinator. I believe you will add the following settings back to your coordinator:

http-server.authentication.type=PASSWORD,KERBEROS
http.server.authentication.krb5.service-name=prestodb
http.server.authentication.krb5.keytab=/etc/prestodb.keytab
http.authentication.krb5.config=/etc/krb5.conf
http-server.https.enabled=true
http-server.https.port=8443
http-server.https.keystore.path=/etc/prestodb.jks
http-server.https.keystore.key=xxxxxxx

Don't add other settings except those mentioned in the doc.

And follow this doc to use the Presto CLI to connect to your Presto coordinator via the HTTPS port.

The extra settings you should also apply to the coordinator is to set up firewall rules to only allow HTTP access from worker nodes. This would depend on what system you are using and what firewall tool you are able to install/use.

[Consultante-yr]

Hello @yhwang,

My coordinator has this following settings:

coordinator=true
discovery-server.enabled=true
node-scheduler.include-coordinator=false

discovery.uri=http://coordinator.domaine.com:8443

http-server.authentication.type=PASSWORD,KERBEROS
http.server.authentication.krb5.service-name=prestodb
http.server.authentication.krb5.keytab=/etc/prestodb.keytab
http.authentication.krb5.config=/etc/krb5.conf
internal-communication.kerberos.enabled=false

http-server.http.enabled=true
http-server.http.port=8443

query.client.timeout=120.00m
query.max-memory=4GB
query.max-memory-per-node=4GB
query.max-total-memory=4GB
query.max-total-memory-per-node=4GB

node-scheduler.network-topology=flat

and in the workers:

discovery.uri=http://coordinator.domaine.com:8443

http-server.http.enabled=true
http-server.http.port=8443

query.client.timeout=120.00m
query.max-memory=2GB
query.max-memory-per-node=1GB
query.max-total-memory=4GB
query.max-total-memory-per-node=1GB

node-scheduler.network-topology=flat
exchange.http-client.request-timeout=20s

node-scheduler.network-topology=flat
exchange.http-client.request-timeout=20s

And, Now if i add these settings to coordinator:
http-server.https.enabled=true
http-server.https.port=8443
http-server.https.keystore.path=/etc/prestodb.jks
http-server.https.keystore.key=xxxxxxx

I get this error: "java.net.BindException: Address already in use", because i have already use the port in http protocol.
So i tried also to put just these settings:
http-server.https.enabled=true
http-server.https.keystore.path=/etc/prestodb.jks
http-server.https.keystore.key=xxxxxxx
But it doesn't work....

For information before disabling the internal kerberos and before changing discovery.ui to http, the Presto CLI works fine.

Thanks in advance,
Best regards,