How to check query level statistics

[wesleygoi-liftoff]

Hi team,

would like to have access to spark application UI to explore statistics similar as those describe in the docs are operation logs
https://kyuubi.readthedocs.io/en/master/monitor/logging.html?highlight=deploy%20mode#operation-logs

In the docs it shows this kind of logging

0: jdbc:hive2://10.242.189.214:2181/> select * from src;
2021-10-27 17:00:19.399 INFO operation.ExecuteStatement: Processing kent's query[fb5f57d2-2b50-4a46-961b-3a5c6a2d2597]: INITIALIZED_STATE -> PENDING_STATE, statement: select * from src
2021-10-27 17:00:19.401 INFO operation.ExecuteStatement: Processing kent's query[fb5f57d2-2b50-4a46-961b-3a5c6a2d2597]: PENDING_STATE -> RUNNING_STATE, statement: select * from src
2021-10-27 17:00:19.400 INFO operation.ExecuteStatement: Processing kent's query[26e169a2-6c06-450a-b758-e577ac673d70]: INITIALIZED_STATE -> PENDING_STATE, statement: select * from src
2021-10-27 17:00:19.401 INFO operation.ExecuteStatement: Processing kent's query[26e169a2-6c06-450a-b758-e577ac673d70]: PENDING_STATE -> RUNNING_STATE, statement: select * from src
2021-10-27 17:00:19.402 INFO operation.ExecuteStatement:
           Spark application name: kyuubi_USER_kent_6d4b5e53-ddd2-420c-b04f-326fb2b17e18
                 application ID: local-1635318669122
                 application web UI: http://10.242.189.214:50250
                 master: local[*]
                 deploy mode: client
                 version: 3.2.0
           Start time: 2021-10-27T15:11:08.416
           User: kent

However when i tail the k8s kyuubi server pod it only shows the following:

2023-08-07 06:02:17.405 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing airflow's query[91fa00c4-463a-4c2c-a673-c7eee0c0ac84]: PENDING_STATE -> RUNNING_STATE, statement:
/* {"app": "dbt", "dbt_version": "1.5.3", "profile_name": "datasci_cvr", "target_name": "cosmos_target", "node_id": "model.datascience.some_table"} */

      insert overwrite datascience.vc_1_day


    select `ts`, `device_id`, `num_views` from vc_1_day__dbt_tmp

how do i get show the application UI to show? application web UI: http://10.242.189.214:50250
i've open up port 4040 for the spark UI but i cannot find anything related to the query

is this because I didnt specify spark.submit.deployMode=cluster?

[yaooqinn]

There is a Kyuubi Query Page tab on Spark UI, and if you go to the Spark History UI, you need to add the spark engine jar to the Spark History classpath.

[wesleygoi-liftoff]

hi @yaooqinn , if we have a successful query and we want to look back into the details eg. task duration GC usage. where can this persisted?

[zwangsheng]

IMO, if your want to find engine detail metrics, such as task metrics, you can find those in engine ui(etc. Spark Engine -> Spark UI/Spark History Server).

[yaooqinn]

You can use the Spark event log feature to persist data and view after the fact. FYI, https://spark.apache.org/docs/3.4.1/monitoring.html

[wesleygoi-liftoff]

hi @yaooqinn , when we set eventlog dir to s3, the driver is still running and will not flush the logs. Have you encountered this?

[yaooqinn]

We do not use s3, I've no idea, sorry