Multiple Import Metadata actions #1178

shreyabiradar07 · 2024-04-23T10:55:12Z

This PR has following changes:

Delete metadata by datasourceName database functionality
Implementation of DELETE /dsmetadata API
curl -X DELETE http://${URL}/dsmetadata -d @./input_datasource.json

Example input_datasource.json (same as that of import dsmetadata request body)

{
    "version":"v1.0",
    "datasource_name":"prometheus-1"
}

shreyabiradar07 · 2024-05-13T05:57:18Z

@msvinaykumar can you please review the PR

msvinaykumar · 2024-05-13T06:39:15Z

@shreyabiradar07 can you start MD file describe above functionality

src/main/java/com/autotune/database/dao/ExperimentDAOImpl.java

msvinaykumar · 2024-05-13T07:06:54Z

src/main/java/com/autotune/database/dao/ExperimentDAO.java

@@ -94,4 +94,6 @@ public interface ExperimentDAO {
    // Load metadata
    List<KruizeDSMetadataEntry> loadMetadata() throws Exception;

+    // Delete metadata
+    public ValidationOutputData deleteKruizeDSMetadataEntryByName(String dataSourceName);


Perhaps you could name it something like "removeClusterMetadataByName" or

Have followed the similar naming convention as deleteKruizeExperimentEntryByName, please confirm if this still requires renaming.

msvinaykumar · 2024-05-13T07:08:37Z

src/main/java/com/autotune/database/dao/ExperimentDAO.java

@@ -94,4 +94,6 @@ public interface ExperimentDAO {
    // Load metadata
    List<KruizeDSMetadataEntry> loadMetadata() throws Exception;


loadClusterMetaData

currently there no usages of loadMetadata function as it fetches all the records from KruizeDSMetadataEntry table (in case when there are multiple datasources will retrieve all the container data not limiting to a specific cluster), hence will remove the unused functionality.

shreyabiradar07 · 2024-05-13T09:49:10Z

@shreyabiradar07 can you start MD file describe above functionality

Documented delete metadata functionality in KruizeLocalAPI.md as part of PR 1174

msvinaykumar

LGTM

msvinaykumar · 2024-06-04T12:06:29Z

UI Flow --> First get if no data then POST and get if still no data then give up (Display error if any)
also Refresh button should be avaible

curl -X post http://${URL}/dsmetadata -d @./input_datasource.json

For above request in backend
Check if data available in db
if yes delete and fetch again from cluster and save it db
if no fetch from cluster and save it db

curl -X post http://${URL}/dsmetadata -d @./input_datasource.json

For above request in backend
Check if data available in db
if yes fetch from cluster and save it db

curl -X get http://${URL}/dsmetadata?datasource_name="abc"
Check if db having data
if yes return data
if no 404 not found

UserWorflow
By default metatdata is prefetched using this URL (This happens as part of kruize pod startup) OR fetch button at UI
curl -X post http://${URL}/dsmetadata -d @./input_datasource.json
there after user will have get operation
curl -X get http://${URL}/dsmetadata?datasource_name="abc"
if user wants to refresh then this api is called
curl -X post http://${URL}/dsmetadata -d @./input_datasource.json

      when user click on cluster -> trigger 
                          curl -X get http://${URL}/dsmetadata?datasource_name="abc"  by default

No need of refresh flag

msvinaykumar · 2024-06-04T12:11:10Z

{
"version":"v1.0",
"datasource_name":"prometheus-1"
}
should have option to provide cluster_name , namespace , workload , container_name

msvinaykumar · 2024-06-06T07:29:25Z

@shreyabiradar07 please set up code walk through session

msvinaykumar

LGTM

dinogun · 2024-06-13T12:09:06Z

src/main/java/com/autotune/analyzer/services/DSMetadataService.java

+                    }
+                }
+
+                new DataSourceManager().importMetadataFromDataSource(datasource);


As mentioned in the review, this can cause DB corruption if for any reason the import fails as we would have removed the DB content. Please reverse the order and add validation to the import results

Updated the code flow including validation of imported DataSourceMetadataInfo object

dinogun · 2024-06-13T12:11:52Z

src/main/java/com/autotune/analyzer/services/DSMetadataService.java

+
+            inputData = request.getReader().lines().collect(Collectors.joining());
+            if (null == inputData || inputData.isEmpty()) {
+                throw new Exception("Request input data cannot be null or empty");


Move all of the strings to string constants

dinogun · 2024-06-13T12:12:59Z

src/main/java/com/autotune/analyzer/services/DSMetadataService.java

                DataSourceMetadataInfo dataSourceMetadata = new ExperimentDBService().loadMetadataFromDBByName(dataSourceName, "false");
+                if (null != dataSourceMetadata) {


Move all of the code that handles the doPost to the relevant files under datasource

src/main/java/com/autotune/analyzer/services/DSMetadataService.java

chandrams · 2024-06-20T12:43:49Z

src/main/java/com/autotune/common/datasource/DataSourceManager.java

@@ -110,6 +110,10 @@ public void deleteMetadataFromDataSource(DataSourceInfo dataSource) {
     */
    public void saveMetadataFromDataSourceToDB(DataSourceInfo dataSourceInfo) {
        try {
+            if (null == dataSourceInfo) {


@shreyabiradar07 - As discussed, please add the negative scenario to bring down the prometheus by scaling down the statefulset and deployment between multiple imports, and validate that the second call to import metadata fails with appropriate error message and list metadata returns the metadata from the DB. This can be added in a separate PR.

shreyabiradar07 added 3 commits April 23, 2024 16:18

Add delete metadata by datasourceName DB method

46ce94c

Revert accidental DB constant push

1b562b2

Implement delete metadata API

a598685

shreyabiradar07 self-assigned this Apr 23, 2024

shreyabiradar07 added local_monitoring kruize-local Tag for mentioning all the PR's and issues raised which covers the kruize local monitoring usecase labels Apr 23, 2024

shreyabiradar07 requested review from dinogun, khansaad and msvinaykumar April 23, 2024 12:27

shreyabiradar07 marked this pull request as ready for review April 23, 2024 12:28

Remove unsed import statements

23025dd

rbadagandi1 added this to the Kruize 0.0.22_rm Release milestone Apr 24, 2024

rbadagandi1 added remote_monitoring and removed local_monitoring labels Apr 24, 2024

rbadagandi1 modified the milestones: Kruize 0.0.22_rm Release, Kruize 0.0.21_rm Release Apr 24, 2024

rbadagandi1 removed the kruize-local Tag for mentioning all the PR's and issues raised which covers the kruize local monitoring usecase label Apr 24, 2024

dinogun modified the milestones: Kruize 0.0.21_rm Release, Kruize 0.0.22_rm Release Apr 24, 2024

rbadagandi1 added kruize-local Tag for mentioning all the PR's and issues raised which covers the kruize local monitoring usecase and removed kruize-local Tag for mentioning all the PR's and issues raised which covers the kruize local monitoring usecase labels Apr 24, 2024

rbadagandi1 modified the milestones: Kruize 0.0.22_rm Release, Kruize 0.0.21_rm Release Apr 24, 2024

dinogun modified the milestones: Kruize 0.0.21_rm Release, Kruize 0.0.22_rm Release Apr 24, 2024

msvinaykumar reviewed May 13, 2024

View reviewed changes

src/main/java/com/autotune/database/dao/ExperimentDAOImpl.java Show resolved Hide resolved

msvinaykumar reviewed May 13, 2024

View reviewed changes

Remove unused loadMetadata functionality

9379994

msvinaykumar approved these changes May 15, 2024

View reviewed changes

dinogun modified the milestones: Kruize 0.0.22_rm Release, Kruize 0.0.23_rm Release May 17, 2024

shreyabiradar07 added 2 commits May 23, 2024 10:20

Merge upstream branch 'mvp_demo' into refactor_import_metadata

901dbaf

Add refresh parameter to import metadata API

70147bd

shreyabiradar07 added 2 commits June 5, 2024 14:24

Revert refresh parameter changes

38bfee3

Error handling using sendErrorResponse instead of throwing exceptions

53f0fcb

msvinaykumar approved these changes Jun 6, 2024

View reviewed changes

dinogun reviewed Jun 13, 2024

View reviewed changes

src/main/java/com/autotune/analyzer/services/DSMetadataService.java Show resolved Hide resolved

shreyabiradar07 added 3 commits June 18, 2024 21:34

Refactor POST /dsmetadata and validate imported metadata object

bfe7cfc

Refactor DELETE /dsmetadata and add java docs

de96117

Add null check for DataSourceInfo

dff11a8

chandrams reviewed Jun 20, 2024

View reviewed changes

Refactor validation logic of metadata

2ed70f8

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multiple Import Metadata actions #1178

Multiple Import Metadata actions #1178

shreyabiradar07 commented Apr 23, 2024 •

edited

shreyabiradar07 commented May 13, 2024

msvinaykumar commented May 13, 2024

msvinaykumar May 13, 2024 •

edited

shreyabiradar07 May 15, 2024

msvinaykumar May 13, 2024

shreyabiradar07 May 15, 2024

shreyabiradar07 commented May 13, 2024

msvinaykumar left a comment

msvinaykumar commented Jun 4, 2024 •

edited

msvinaykumar commented Jun 4, 2024

msvinaykumar commented Jun 6, 2024

msvinaykumar left a comment

dinogun Jun 13, 2024

shreyabiradar07 Jun 21, 2024

dinogun Jun 13, 2024

shreyabiradar07 Jun 21, 2024

dinogun Jun 13, 2024

chandrams Jun 20, 2024

		@@ -94,4 +94,6 @@ public interface ExperimentDAO {
		// Load metadata
		List<KruizeDSMetadataEntry> loadMetadata() throws Exception;

		DataSourceMetadataInfo dataSourceMetadata = new ExperimentDBService().loadMetadataFromDBByName(dataSourceName, "false");
		if (null != dataSourceMetadata) {

Multiple Import Metadata actions #1178

Are you sure you want to change the base?

Multiple Import Metadata actions #1178

Conversation

shreyabiradar07 commented Apr 23, 2024 • edited

shreyabiradar07 commented May 13, 2024

msvinaykumar commented May 13, 2024

msvinaykumar May 13, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shreyabiradar07 commented May 13, 2024

msvinaykumar left a comment

Choose a reason for hiding this comment

msvinaykumar commented Jun 4, 2024 • edited

msvinaykumar commented Jun 4, 2024

msvinaykumar commented Jun 6, 2024

msvinaykumar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shreyabiradar07 commented Apr 23, 2024 •

edited

msvinaykumar May 13, 2024 •

edited

msvinaykumar commented Jun 4, 2024 •

edited