List and delete policy recommendation jobs

[dreamtalen]

Add a list command to show all policy recommendation jobs, e.g:

> theia pr list
CreateTime          CompleteTime        ID                                   Status
2022-06-17 18:33:15 N/A                 2cf13427-cbe5-454c-b9d3-e1124af7baa2 RUNNING
2022-06-17 18:06:56 2022-06-17 18:08:37 69e2e543-60e9-4d45-97a1-d56337966579 COMPLETED
2022-06-16 23:41:43 2022-06-16 23:43:15 a65daf22-8e7e-4479-9f4e-edc1d99716ff COMPLETED
N/A                 2022-06-13 22:19:17 749ecc41-bf5e-4d08-88ef-fb66b60bf1fb COMPLETED
N/A                 2022-06-15 21:41:16 1e7ffc6d-2321-422d-b982-0ffca2d7987f COMPLETED

We will fetch and display all sparkapplication from the K8s API server first, then we will check the recommendation result table of ClickHouse DB for additional completed jobs that are not inside the K8s API server (probably they are deleted by users manually).
CompleteTime of uncompleted jobs will show as N/A, CreateTime of jobs fetching from the result table of ClickHouse DB will show as N/A since we didn't save that info in DB.

Add a delete command to let users delete a policy recommendation job given ID.

If the status of this job is not completed(like running, falied, etc), the sparkapplication behind this job will be deleted. Otherwise, both sparkapplication behind this job and the recommendation result in database will be deleted. e.g:

> theia pr delete 1e7ffc6d-2321-422d-b982-0ffca2d7987f
Successfully deleted policy recommendation job with ID 1e7ffc6d-2321-422d-b982-0ffca2d7987f

Open question:

1. Do we need to add a status column in the recommendations table?
   Currently, statuses of policy recommendation jobs are obtained from the k8s API server. Only completed jobs will write results into the recommendations table.
   May need to think about how to sync status between the k8s API server and database. (Should be handled in the middle layer application later)
   For now, let's not add more columns to the recommendation result table.

2. Do we have other columns that would like to be added to the list command result?
   job parameters, flows number, etc.
   (For now we don't plan to show job parameters in list command since they are too much to display in a table, users could see them by describe the driver pod)

3. Check the failure reason for failed jobs and write them into database
   Could check the debug APIs of spark operator first.
   There is a Error Message field in failed sparkapplication, we could display it in the status command of cli.
   Writing it into database should be handled in the middle layer application later.

[yuntanghsu]

Will the completed jobs be deleted in the K8s API server?

I think we can also add the process-time to the recommendations table.

[dreamtalen]

Will the completed jobs be deleted in the K8s API server?

No, unless the user deletes it through theia cli or api server.

I think we can also add the process-time to the recommendations table.

Sure, it will be useful to have process time in DB. But it will have the same situation with status column, we need the middle layer application to update and sync this info with the API server, so we will do this later probably.