[BUG] Opensearch-dashboards Kerberos issue

[alexbekk]

I configured the kerberos as described in the documentation and the access to the nodes works as it should. But, it doesn't work for opensearch-dashboards. If I specify in opensearch-dashboards.yml the line:

opensearch_security.auth.type: "kerberos"

then the cluster crashes.
opensearch-dashboards | {"type": "log","@timestamp": "2022-02-03T12:34:14Z", "tags":["fatal", "root"], "pid":1, "message": "Error: Unsupported authentication type: kerberos\n at getAuthenticationHandler (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/auth/auth_handler_factory. ts:75:13)\n at SecurityPlugin.setup (/usr/share/opensearch-dashboards/plugins/securityDashboards/server/plugin.ts:110:39)"}
opensearch-dashboards | {"type":"log","@timestamp":"2022-02-03T12:34:14Z","tags":["info","plugins-system"],"pid":1,"message":"Stopping all plugins."}
opensearch-dashboards |
opensearch-dashboards | FATAL Error: Unsupported authentication type: kerberos
opensearch-dashboards |
opensearch-dashboards exited with code 1

On the other hand, if I delete this line, I get this error from dashboards:
opensearch-node2 | [2022-02-03T12:43:21,465][WARN ][c.a.d.a.h.k.HTTPSpnegoAuthenticator] [opensearch-node2] No 'Negotiate Authorization' header, send 401 and 'WWW-Authenticate Negotiate'
opensearch-dashboards | {"type":"log","@timestamp":"2022-02-03T12:43:21Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}

And dashboards goes down.

Does opensearch-dashboards support kerberos and what additional settings need to be made?

config.yml

_meta:
type: "config"
config_version: 2

config:
dynamic:
http:
anonymous_auth_enabled: false
authc:
internal_auth:
order: 1
description: "HTTP basic authentication using the internal user database"
http_enabled: true
transport_enabled: true
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: internal
ldap_auth:
order: 2
description: "Authenticate using LDAP"
http_enabled: false
transport_enabled: false
http_authenticator:
type: basic
challenge: false
authentication_backend:
type: ldap
config:
pemtrustedcas_filepath: /usr/share/opensearch/config/ca.pem
enable_ssl: true
enable_start_tls: false
enable_ssl_client_auth: false
verify_hostnames: false
hosts:
- x.x.x.x:636
bind_dn: administrator@domain.ad
password: password
userbase: cn=Users,dc=domain,dc=ad
usersearch: (cn={0})
username_attribute: cn
kerberos_auth_domain:
http_enabled: true
order: 0
http_authenticator:
type: kerberos
challenge: true
config:
krb_debug: true
strip_realm_from_principal: true
authentication_backend:
type: noop

authz:
  ldap_roles:
    description: "Authorize using LDAP"
    http_enabled: true
    transport_enabled: true
    authorization_backend:
      type: ldap
      config:
        pemtrustedcas_filepath: /usr/share/opensearch/config/ca.pem
        enable_ssl: true
        enable_start_tls: false
        enable_ssl_client_auth: false
        verify_hostnames: false
        hosts:
        - x.x.x.x:636
        bind_dn: administrator@domain.ad
        password: password
        userbase: cn=Users,dc=domain,dc=ad
        usersearch: (cn={0})
        username_attribute: cn
        skip_users:
          - admin
          - kibanaserver
        rolebase: ou=Groups,dc=domain,dc=ad
        rolesearch: (Member={0})
        userroleattribute: null
        userrolename: disabled
        rolename: cn
        resolve_nested_roles: true

opensearch-dashboards.yml

opensearch.ssl.verificationMode: certificate
server.host: 0.0.0.0
opensearch.username: "kibanaserver"
opensearch.password: "kibanaserver"
opensearch.requestHeadersWhitelist: [ authorization,securitytenant ]
server.ssl.enabled: true
server.ssl.certificate: /usr/share/opensearch-dashboards/config/client.pem
server.ssl.key: /usr/share/opensearch-dashboards/config/client-key.pem
opensearch.ssl.certificateAuthorities: [ "/usr/share/opensearch-dashboards/config/root-ca.pem"]
opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: ["Private", "Global"]
opensearch_security.readonly_mode.roles: ["kibana_read_only"]
opensearch_security.cookie.secure: true

opensearch.yml

plugins.security.ssl.transport.pemcert_filepath: node1.pem
plugins.security.ssl.transport.pemkey_filepath: node1-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: node1.pem
plugins.security.ssl.http.pemkey_filepath: node1-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:

• 'CN=ADMIN,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
  plugins.security.nodes_dn:
• 'CN=node1.example.com,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
• 'CN=node2.example.com,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
  plugins.security.audit.type: internal_opensearch
  plugins.security.enable_snapshot_restore_privilege: true
  plugins.security.check_snapshot_restore_write_privileges: true
  plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
  cluster.routing.allocation.disk.threshold_enabled: false
  opendistro_security.audit.config.disabled_rest_categories: NONE
  opendistro_security.audit.config.disabled_transport_categories: NONE
  plugins.security.kerberos.krb5_filepath: /etc/krb5.conf
  plugins.security.kerberos.acceptor_keytab_filepath: /usr/share/opensearch/config/opensearch-realm.keytab
  plugins.security.kerberos.acceptor_principal: 'HTTP/opensearch.realm.io@REALM.IO'

[stockholmux]

I think this referring to what is outlined here: https://opensearch.org/docs/latest/security-plugin/configuration/configuration/#kerberos

[alexbekk]

I think this referring to what is outlined here: https://opensearch.org/docs/latest/security-plugin/configuration/configuration/#kerberos

Yes, but it's only about opensearch and not a word about dashboards.

[stockholmux]

@alexbekk Ah. Good catch.

@Naarcha-AWS thoughts?

[Naarcha-AWS]

@stockholmux: Does Dashboards require a separate security configuration for kerberos? Correct me if I'm wrong, but any auth we support in OpenSearch should work in Dashboards.

[alexbekk]

@stockholmux: Does Dashboards require a separate security configuration for kerberos? Correct me if I'm wrong, but any auth we support in OpenSearch should work in Dashboards.

Why then can I authenticate to opensearch while dashboards crash?

opensearch-node2 | [2022-02-03T12:43:21,465][WARN ][c.a.d.a.h.k.HTTPSpnegoAuthenticator] [opensearch-node2] No 'Negotiate Authorization' header, send 401 and 'WWW-Authenticate Negotiate'
opensearch-dashboards | {"type":"log","@timestamp":"2022-02-03T12:43:21Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}

[Naarcha-AWS]

@alexbekk: How are you running OpenSearch? Docker, Tarbarll?

[alexbekk]

@Naarcha-AWS docker

[Naarcha-AWS]

@alexbekk: Have you tried stopping the cluster and removing your current volumes?

[alexbekk]

@Naarcha-AWS yep, docker-compose down -v
docker-compose up

[Naarcha-AWS]

@alexbekk: After consulting some developers of the security plugin, it seems like it might be an issue with either the Kerberos replay cache or your negotiate token within Kerberos. Namely, in order for Dashboards to work, you need to disable to Kerberos replay cache. Each auth request requires a unique Keberos ticket. Dashboards will reuse tokens from requests.

To disable the replay cache, in your Kerberos node, add this line to jvm.options,
-Dsun.security.krb5.rcache=none.

After that's complete, try seeing if you can access OpenSearch using curl with the --negotiate option.

curl -XGET --insecure --negotiate -u 'credentials' 'host'

Let me know if this helps.

[melokk97]

@Naarcha-AWS We tried this, but it doesn't work. Are there any other updates on this issue?

[alexbekk]

Command: curl -XGET --insecure --negotiate -u : http://opensearch.adrealm.io:9200
Result before kinit:
{"error":{"header":{"WWW-Authenticate":"Negotiate"}}}
Result after kinit:
{
"name" : "opensearch-node1",
"cluster_name" : "opensearch-cluster",
"cluster_uuid" : "INYQ7r-ZRV6ZLXfrl2WCaQ",
"version" : {
"distribution" : "opensearch",
"number" : "1.2.4",
"build_type" : "tar",
"build_hash" : "e505b10357c03ae8d26d675172402f2f2144ef0f",
"build_date" : "2022-01-14T03:38:06.881862Z",
"build_snapshot" : false,
"lucene_version" : "8.10.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "The OpenSearch Project: https://opensearch.org/"
}

But after curl -XGET --insecure --negotiate -u : http://opensearch.adrealm.io:5601
result: OpenSearch Dashboards server is not ready yet

stdout:
opensearch-node2 | [2022-02-18T13:10:09,922][WARN ][c.a.d.a.h.k.HTTPSpnegoAuthenticator] [opensearch-node2] No 'Negotiate Authorization' header, send 401 and 'WWW-Authenticate Negotiate'
opensearch-dashboards | {"type":"log","@timestamp":"2022-02-18T13:10:09Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}

[Naarcha-AWS]

@alexbekk and @melokk97: I've looked into the issue further and it turns out that Kerberos authentication was missed for the Dashboards security plugin.

To get it implemented, I've created this issue in the security plugin repo: opensearch-project/security-dashboards-plugin#907

For the docs, I'll add a note to the Kerberos section explaining the lack of support.