Got error when posting admin get engine REST with sharelevel=GROUP

[HaoYang670]

Hi Kyuubi community,

I have created 2 Spark-cluster with same user remziy. The former is created in USER share level and the latter is created in GROUP share level (primary_group = "group_remziy"). I can get the info of the former using the admin get engine REST. On the other hand, I got the error KeeperErrorCode = NoNode for /kyuubi_1.7.0_GROUP_SPARK_SQL/remziy when getting access to the latter. Any idea of how to fix it?

[HaoYang670]

@zwangsheng Could you please take a look if you have time? Thanks a lot.

[zwangsheng]

Would you like show some params, such like Kyuubi Server Configurations about ShareLevel or kyuubi.engine.share.level.subdomain?

BTW, you should parse rest url with request params for giving more info.

Also, if GROUP level spark engine had been created, you can find engine registed path from spark driver log.

[HaoYang670]

Would you like show some params?

Sure. We set the sharelevel in the session properties of JDBC URL and leave the subdomain as default.

BTW, you should parse rest url with request params for giving more info.

The rest URL we send is http://{{KYUUBI_HOST}}:{{REST_PORT}}/api/v1/admin/engine?sharelevel=GROUP. And we have updated the authentication header to match the username of the kyuubi engine.

[HaoYang670]

BTW, I can get the handlers of kyuubi sessions of the cluster, and get the info of the kyuubi session.

{
    "sessionId": "*",
    "clientVersion": 9,
    "sessionType": "INTERACTIVE",
    "sessionName": "",
    "user": "remziy",
    "clientIP": "10.244.0.1",
    "serverIP": "0.0.0.0:10009",
    "conf": {
        "kyuubi.session.real.user": "remziy",
        "set:hiveconf:spark.executor.cores": "1",
        "set:hiveconf:spark.hadoop.fs.s3a.secret.key": "*",
        "set:hiveconf:spark.executor.memory": "512m",
        "kyuubi.client.version": "1.7.0",
        "use:database": "default",
        "set:hiveconf:spark.kubernetes.container.image": "*",
        "set:hiveconf:kyuubi.engine.share.level": "GROUP",
        "set:hiveconf:spark.executor.instances": "1",
        "kyuubi.client.ipAddress": "*",
        "set:hiveconf:hive.server2.thrift.resultset.default.fetch.size": "1000",
        "set:hiveconf:spark.hadoop.fs.s3a.access.key": "*",
        "kyuubi.session.connection.url": "0.0.0.0:10009",
        "kyuubi.server.ipAddress": "0.0.0.0",
        "set:hiveconf:spark.kubernetes.namespace": "default"
    },
    "eventTime": 1681202618402,
    "startTime": 1681202618401,
    "remoteSessionId": "*",
    "engineId": "*",
    "openedTime": 1681202618415,
    "endTime": -1,
    "totalOperations": 0,
    "exception": null,
    "eventType": "kyuubi_session"
}

[zwangsheng]

Can you provide spark driver log about Registered EngineServiceDiscovery in namespace ${_namespace}, which will help find where sparkSqlEngine registered.

[zwangsheng]

I think for GROUP level, we should use
/serverSpace_version_GROUP_engineType/user[/subdomain]
right?

Yes

In AdminResource, will find GROUP LEVEL engine in /serverSpace_version_GROUP_engineType/user, but engine register self in /serverSpace_version_GROUP_engineType/prime_user

We should keep AdminResource#getEngineSpace logic same with EngineRef#engineSpace

[HaoYang670]

The admin RESTful API doesn't convert my username to the primary group name, which is why I always got KeeperNoNodeError

[HaoYang670]

In AdminResource, will find GROUP LEVEL engine in /serverSpace_version_GROUP_engineType/user, but engine register self in /serverSpace_version_GROUP_engineType/prime_user

I think this is a bug, right?

[zwangsheng]

In AdminResource, will find GROUP LEVEL engine in /serverSpace_version_GROUP_engineType/user, but engine register self in /serverSpace_version_GROUP_engineType/prime_user

I think this is a bug, right?

Yes, but we should keep as /serverSpace_version_GROUP_engineType/prime_user, so we should modify AdminResource#getEngineSpace

[HaoYang670]

File an issue to track it: #4738
We can move the discussion there