Elastic security/auth issue

[jonnywray]

Hi

trying to get up an instance using Elastic as the graph service but getting the following exception in the GMS pod.

Caused by: org.elasticsearch.ElasticsearchStatusException: method [HEAD], host [http://datahub-elastic-es-http:9200], URI [/graph_service_v1?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=false], status line [HTTP/1.1 401 Unauthorized]

my global values for Elastic are below, and addition of the auth section was enough to get the elastic start up job to work.

global:
      graph_service_impl: elasticsearch

      elasticsearch:
        host: "datahub-elastic-es-http"
        port: "9200"
        auth:
          username: elastic
          password:
            secretRef: datahub-elastic-es-elastic-user
            secretKey: elastic

Any thoughts?
Thanks a lot

Jonny

[jonnywray]

i should have mentioned this is an elastic cluster not created by the prerequisites. We use the elastic operator for managing ES clusters in k8s.

[jonnywray]

Looks like it isn't graph endpoint specific. I added the Neo4j graph impl and tried that, and now getting a similar error on a different endpoint

Caused by: org.elasticsearch.ElasticsearchStatusException: method [HEAD], host [http://datahub-elastic-es-http:9200], URI [/mlmodeldeploymentindex_v2?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=false], status line [HTTP/1.1 401 Unauthorized]

am I doing something wrong in configuring the elastic auth? Or not supported currently?

thanks
Jonny

[dexter-mh-lee]

Sorry about the late message @jonnywray We somehow missed these issues. Do you have RBAC set up in your elasticsearch cluster?

[sergialonsaco]

Hey @dexter-mh-lee! any news on this issue? facing the same situation here

[jonnywray]

Apologies for not replying, I didn't see the reply from last Sept. I ended up using the prerequisites to avoid the problem.

Longer term we would like to use an existing ES cluster. We provision our clusters using the Elastic operator (https://www.elastic.co/guide/en/cloud-on-k8s/current/index.html) and don't do anything special with respect to RBAC. So whatever controls the operator adds by default are used.

[sergialonsaco]

Hey @jonnywray , let me share some logs to see if we're facing the same issue.
In my case, when it starts to run the datahub-gms application it appears to connect:

+ exec dockerize -wait http://elasticsearch-es-http.elastic-data:9200 -wait-http-header 'Authorization:Basic XXXX' -wait tcp://prerequisites-mysql:3306 -wait tcp://kafka-headless.kafka:9092 -timeout 240s java -jar /jetty-runner.jar --jar jetty-util.jar --jar jetty-jmx.jar --config /datahub/datahub-gms/scripts/jetty.xml /datahub/datahub-gms/bin/war.war
+ Waiting for: http://elasticsearch-es-http.elastic-data:9200
+ Received 200 from http://elasticsearch-es-http.elastic-data:9200

And just exactly after this lines, the same error as you happens:
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'updateIndicesHook' defined in URL [jar:file:/tmp/jetty-0_0_0_0-8080-war_war-_-any-4045167199054543407.dir/webapp/WEB-INF/lib/mae-consumer.jar!/com/linkedin/metadata/kafka/hook/UpdateIndicesHook.class]: Bean instantiation via constructor failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.linkedin.metadata.kafka.hook.UpdateIndicesHook]: Constructor threw exception; nested exception is ElasticsearchStatusException[method [HEAD], host [http://elasticsearch-es-http.elastic-data:9200], URI [/graph_service_v1?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=false], status line [HTTP/1.1 401 Unauthorized]]; nested: ResponseException[method [HEAD], host [http://elasticsearch-es-http.elastic-data:9200], URI [/graph_service_v1?ignore_throttled=false&ignore_unavailable=false&expand_wildcards=open%2Cclosed&allow_no_indices=false], status line [HTTP/1.1 401 Unauthorized]];

Also, from the same helm, I wanna add that the elasticsearchSetupJob is working perfectly fine. So it looks like is an internal issue from the datahub-gms code?
FYI @dexter-mh-lee

[github-actions]

This issue is stale because it has been open for 30 days with no activity. If you believe this is still an issue on the latest DataHub release please leave a comment with the version that you tested it with. If this is a question/discussion please head to https://slack.datahubproject.io. For feature requests please use https://feature-requests.datahubproject.io

[github-actions]

This issue was closed because it has been inactive for 30 days since being marked as stale.