autoOptimize enabled #902

sriram251-code · 2023-04-24T14:40:20Z

closes #850

PR proof added in the Azure workspace.

GeekSheikh · 2023-04-28T14:43:53Z

@sriram251-code I still don't see a proof for this or a requested peer review for this PR. Please get this taken care of and I will review once these are complete.

GeekSheikh

couple of safety valves needed plus a few comments on semantics.

GeekSheikh · 2023-05-05T13:36:15Z

src/main/scala/com/databricks/labs/overwatch/env/Database.scala

+        if (!isAutoOptimise) {
+          val alterStatement = s"""alter table ${target.tableFullName} set TBLPROPERTIES (delta.autoOptimize.optimizeWrite = true)"""
+          logger.log(Level.INFO, alterStatement)
+          if (config.debugFlag) println(alterStatement)
+          spark.sql(alterStatement) //Turning autoOptimize for the table.
+        }


isAutoOptimize val name is confusing -- we have two conditions -- is the target set to autoOptimize and we have is the target actually autoOptimize -- I think we need to improve the val names to make this more clear. Also add some inline code comments to the if statements.

maybe something like targetPropertiesAccurate

GeekSheikh · 2023-05-05T13:36:33Z

src/main/scala/com/databricks/labs/overwatch/env/Database.scala

+   * @param target
+   * @return
+   */
+  private def checkAutoOptimiseWrite(target: PipelineTable): Boolean = {


optimize in American English is spelled with Z btw.

GeekSheikh · 2023-05-05T13:47:27Z

src/main/scala/com/databricks/labs/overwatch/env/Database.scala

+   */
+  private def checkAutoOptimiseWrite(target: PipelineTable): Boolean = {
+    val deltaProperties = spark.sql(s"""describe detail '${target.tableLocation}'""").select("properties").head().get(0).asInstanceOf[Map[String, Boolean]]
+    val optimiseWrite: Boolean = deltaProperties.get("delta.autoOptimize.optimizeWrite").getOrElse("true").toString.toBoolean


don't use .get...getOrElse this should all be a single command .getOrElse

Why are you defaulting to true?

GeekSheikh · 2023-05-05T13:49:54Z

src/main/scala/com/databricks/labs/overwatch/env/Database.scala

+   * @return
+   */
+  private def checkAutoOptimiseWrite(target: PipelineTable): Boolean = {
+    val deltaProperties = spark.sql(s"""describe detail '${target.tableLocation}'""").select("properties").head().get(0).asInstanceOf[Map[String, Boolean]]


never use .head() I know we should never get empty return here but for safety use .headOption().getOrElse(throw Exception...) to avoid None.get errors.

What if one of the table properties is not a string=boolean but string=string? This will fail right? The actual data type is map[string, string] so don't try to collect and force cast it as something else.

try something like this instead.

spark.sql("describe detail overwatch_global_etl.sparkTask_gold").select('properties).as[Map[String, String]].collect().headOption.getOrElse(throw new Exception("string")).getOrElse("delta.autoOptimize.optimizeWrite", "false").toBoolean

GeekSheikh · 2023-05-05T14:02:10Z

src/main/scala/com/databricks/labs/overwatch/env/Database.scala

+    optimiseWrite
+  }
+
+
  /**
   * register an Overwatch target table in the configured Overwatch deployment


I think we need to add some more comments to explain the reasoning behind recreating as it was then altering. I doubt most people understand that there will be a failure if you try to create with diff property than the files have.

That said, what happens if you try to alter at the file level delta.filepath and the table -- does that work or does that still fail?

If u alter the the file level delta then it will also work.
Comments updated.

GeekSheikh

I think you need to refactor the entire registerTarget function to handle all the cases. Perhaps you should first write down all the cases that are to be handled. Refactor this into a few functions if necessary to handle all the scenarios. Develop sufficient tests to ensure that all scenarios are handled.

Please update the function definition and improve your proof to account for the scenarios you intend to handle.

GeekSheikh · 2023-05-18T19:55:18Z

build.sbt

@@ -2,7 +2,7 @@ name := "overwatch"

 organization := "com.databricks.labs"

-version := "0.7.2.0"
+version := "0.7.2.0.1"


this is the wrong version. Please be sure to rebase onto 0721_release branch.

GeekSheikh · 2023-05-18T20:00:12Z

src/main/scala/com/databricks/labs/overwatch/env/Database.scala

+   */
+  private def checkAutoOptimizeWrite(target: PipelineTable): Boolean = {
+    import spark.implicits._
+    val optimiseWrite: Boolean = spark.sql(s"""describe detail '${target.tableLocation}'""")


sp -- optimizeWrite

GeekSheikh · 2023-05-18T20:00:49Z

src/main/scala/com/databricks/labs/overwatch/env/Database.scala

+    import spark.implicits._
+    val optimiseWrite: Boolean = spark.sql(s"""describe detail '${target.tableLocation}'""")
+      .select("properties").as[Map[String, String]].collect()
+      .headOption.getOrElse(throw new Exception("string"))


put a valid exception string here not string

GeekSheikh · 2023-05-18T20:01:54Z

src/main/scala/com/databricks/labs/overwatch/env/Database.scala

+
+  /**
+   * register an Overwatch target table in the configured Overwatch deployment.
+   * After release > 7.2.1 : When the tables are dropped ,it will create the table as it was previously created then check whether it has autoOptimize enabled or not.


line is too long, create line break

GeekSheikh · 2023-05-18T20:02:27Z

src/main/scala/com/databricks/labs/overwatch/env/Database.scala

+
+  /**
+   * register an Overwatch target table in the configured Overwatch deployment.
+   * After release > 7.2.1 : When the tables are dropped ,it will create the table as it was previously created then check whether it has autoOptimize enabled or not.


As of 0.7.2.1 -- not after 0721

GeekSheikh · 2023-05-18T20:09:48Z

src/main/scala/com/databricks/labs/overwatch/env/Database.scala

+          spark.sql(alterStatement) //Turning autoOptimize for the table.
+        }
+
+      } else {


Please add a comment to this else

} else { // target is not auto optimize

GeekSheikh · 2023-05-18T20:10:36Z

src/main/scala/com/databricks/labs/overwatch/env/Database.scala

+        val targetPropertiesAccurate = checkAutoOptimizeWrite(target)
+        val createStatement = s"create table if not exists ${target.tableFullName} " +
+          s"USING DELTA location '${target.tableLocation}' " +
+          s"TBLPROPERTIES (delta.autoOptimize.optimizeWrite = ${targetPropertiesAccurate},delta.autoOptimize.autoCompact=false)"


you should force this boolean to a string

${targetPropertiesAccurate.toString}

GeekSheikh · 2023-05-18T20:12:22Z

src/main/scala/com/databricks/labs/overwatch/env/Database.scala

+        if (config.debugFlag) println(logMessage)
+        spark.sql(createStatement) //Creating the table as it was previously created.
+        if (!targetPropertiesAccurate) {
+          val alterStatement = s"""alter table ${target.tableFullName} set TBLPROPERTIES (delta.autoOptimize.optimizeWrite = true)"""


this should not ALWAYS be set to true if !targetPropertiesAccurate. What if optimizeWrite in the table is true but it's definition defines that it should be false?

I realize that you're in an if block to handle only target.autoOptimize but nonetheless, the code is confusing and you don't handle if the opposite is true (config == false and actual == true)

GeekSheikh · 2023-05-18T20:13:31Z

src/main/scala/com/databricks/labs/overwatch/env/Database.scala

+  private def checkAutoOptimizeWrite(target: PipelineTable): Boolean = {
+    import spark.implicits._
+    val optimiseWrite: Boolean = spark.sql(s"""describe detail '${target.tableLocation}'""")
+      .select("properties").as[Map[String, String]].collect()
+      .headOption.getOrElse(throw new Exception("string"))
+      .getOrElse("delta.autoOptimize.optimizeWrite", "false")
+      .toBoolean
+    optimiseWrite
+  }


seems like this function should not return whether or not the property is true or false but rather whether or not it matches the target config, right?

shouldn't it return
optimizeWrite == target.autoOptimize?

I think you're trying to get too fancy with what this function does. Perhaps two functions would be easier to follow.

Is the target actually optimizeWrite

Does the target actual setting match it's config

GeekSheikh · 2023-05-18T20:19:04Z

src/main/scala/com/databricks/labs/overwatch/env/Database.scala

+        if (!targetPropertiesAccurate) {
+          val alterStatement = s"""alter table ${target.tableFullName} set TBLPROPERTIES (delta.autoOptimize.optimizeWrite = true)"""


Since you're inside an if block of !target.exists(catalogValidation = true) then this will never work because the table does not exist once you get to this block of code since alter table tableName will result in Table Does not Exist.

sonarcloud · 2023-05-22T13:33:54Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

sonarcloud · 2023-06-27T17:32:30Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

* autoOptimize enabled * review comments implemented * reordered function calls * updated func name and comments --------- Co-authored-by: Daniel Tomes <10840635+GeekSheikh@users.noreply.github.com>

* Initial commit * Snapshot Incremental * Snapshot Incremental * Snapshot Incremental * Snapshot Incremental * Verbose Audit Logging * Incremental Snapshot * Incremental Snapshot * ETL Migration * Snapshot Incremental * Snapshot Incremental * Snapshot Incremental * Snapshot Incremental * Added Restore Logic * Add remoteWorkspaceByPath in Migration script * Add Changes * Add Changes * Add Changes * Add Changes * Add Changes * Add Changes * Add Changes * Added changes as per comment from Guinea * Change as per latest PR comments * Change as per latest PR comments * Change as per latest PR comments * added support for delta table in updateConfig * Change the flow in Migration. Drop Database should be done after updateConfig * Change the flow in Migration. Drop Database should be done after updateConfig * Initial commit * autoOptimize enabled (#902) * autoOptimize enabled * review comments implemented * reordered function calls * updated func name and comments --------- Co-authored-by: Daniel Tomes <10840635+GeekSheikh@users.noreply.github.com> * Ambiguous API call error msg fixed (#944) * Initial commit * code fix --------- Co-authored-by: Guenia Izquierdo <guenia.izquierdo@databricks.com> * Snapshot Incremental * Snapshot Incremental * Snapshot Incremental * Snapshot Incremental * Verbose Audit Logging * Incremental Snapshot * Incremental Snapshot * ETL Migration * Snapshot Incremental * Snapshot Incremental * Snapshot Incremental * Snapshot Incremental * Added Restore Logic * Add remoteWorkspaceByPath in Migration script * Add Changes * Add Changes * Add Changes * Add Changes * Add Changes * Add Changes * Add Changes * Added changes as per comment from Guinea * Change as per latest PR comments * Change as per latest PR comments * Change as per latest PR comments * added support for delta table in updateConfig * Change the flow in Migration. Drop Database should be done after updateConfig * Change the flow in Migration. Drop Database should be done after updateConfig * Omit Main Method * Omit Main Method * Omit Main Method * Omit Main Method * Added Runner for both Migration.scala and Snapshot.scala * Added Runner for Restore.scala * Added Runner for Restore.scala * Added Runner for Restore.scala * Change the writeStream function Name * Added Validation for Snapshot.scala * Added Validation for Snapshot.scala * Added Validation for Snapshot.scala * Added Validation for Migration.scala and Restore.scala * Added Validation for Migration.scala and Restore.scala * Added Validation for Migration.scala and Restore.scala * Added Validation for Migration.scala and Restore.scala * Added Validation for Migration.scala and Restore.scala * Add Verify Min Schema in audit_log_bronze * Add Verify Min Schema in audit_log_bronze * Changed Apply function to Process * Change the Runner script to run as per the new process function * Change the Runner script to run as per the new process function * Change the Runner script to run as per the new process function * Change the Runner script to run as per the new process function * Removed println for Zone Validation * Merge 721 * Changed the negative condition in Snapshot.scala * Resolve the Conflict in Database.scala --------- Co-authored-by: Guenia Izquierdo <guenia.izquierdo@databricks.com> Co-authored-by: Sourav Banerjee <30810740+Sourav692@users.noreply.github.com> Co-authored-by: Sriram Mohanty <69749553+sriram251-code@users.noreply.github.com> Co-authored-by: Daniel Tomes <10840635+GeekSheikh@users.noreply.github.com>

* Initial commit * autoOptimize enabled (#902) * autoOptimize enabled * review comments implemented * reordered function calls * updated func name and comments --------- Co-authored-by: Daniel Tomes <10840635+GeekSheikh@users.noreply.github.com> * Ambiguous API call error msg fixed (#944) * Initial commit * code fix --------- Co-authored-by: Guenia Izquierdo <guenia.izquierdo@databricks.com> * 931 shuffle factor -- mega --------- Co-authored-by: Guenia Izquierdo <guenia.izquierdo@databricks.com> Co-authored-by: Sriram Mohanty <69749553+sriram251-code@users.noreply.github.com>

* autoOptimize enabled * review comments implemented * reordered function calls * updated func name and comments --------- Co-authored-by: Daniel Tomes <10840635+GeekSheikh@users.noreply.github.com>

* autoOptimize enabled (#902) * autoOptimize enabled * review comments implemented * reordered function calls * updated func name and comments --------- Co-authored-by: Daniel Tomes <10840635+GeekSheikh@users.noreply.github.com> * Update_Azure_gcp_instances Update_Azure_gcp_instances --------- Co-authored-by: Sriram Mohanty <69749553+sriram251-code@users.noreply.github.com> Co-authored-by: Daniel Tomes <10840635+GeekSheikh@users.noreply.github.com>

* Initial commit * autoOptimize enabled (#902) * autoOptimize enabled * review comments implemented * reordered function calls * updated func name and comments --------- Co-authored-by: Daniel Tomes <10840635+GeekSheikh@users.noreply.github.com> * Ambiguous API call error msg fixed (#944) * Initial commit * code fix --------- Co-authored-by: Guenia Izquierdo <guenia.izquierdo@databricks.com> * 981 - Remove assertion and replace with warning * 743 - improved supported scenarios for rollbackPipelineForModule --------- Co-authored-by: Guenia Izquierdo <guenia.izquierdo@databricks.com> Co-authored-by: Sriram Mohanty <69749553+sriram251-code@users.noreply.github.com>

* Initial commit * Snapshot Incremental * Snapshot Incremental * Verbose Audit Logging * Verbose Audit Logging * Verbose Audit Logging * Add Changes * Add Changes * Add Changes * Add Changes * Add Changes * Add Changes * Initial commit * autoOptimize enabled (#902) * autoOptimize enabled * review comments implemented * reordered function calls * updated func name and comments --------- Co-authored-by: Daniel Tomes <10840635+GeekSheikh@users.noreply.github.com> * Ambiguous API call error msg fixed (#944) * Initial commit * code fix --------- Co-authored-by: Guenia Izquierdo <guenia.izquierdo@databricks.com> * Snapshot Incremental * Snapshot Incremental * Verbose Audit Logging * Verbose Audit Logging * Verbose Audit Logging * Add Changes * Add Changes * Add Changes * Add Changes * Add Changes * Add Changes * add as per comment * Change CLSF impute logic * Change CLSF impute logic * Resolve the Conflict in Database.scala * Resolve the Conflict in Database.scala * Resolve the Conflict in Database.scala * Resolve the Conflict in Database.scala * Resolve the Conflict in Database.scala * Change Azure_instance_Detail.csv * Change Azure_instance_Detail.csv * Change Azure_instance_Detail.csv * Change Azure_instance_Detail.csv * Change Window function to GroupBy in buildNotebookCommandsFact * Rollback the groupBy again to window function. * Change the window functions name in buildNotebookCommandsFact --------- Co-authored-by: Guenia Izquierdo <guenia.izquierdo@databricks.com> Co-authored-by: Sourav Banerjee <30810740+Sourav692@users.noreply.github.com> Co-authored-by: Sriram Mohanty <69749553+sriram251-code@users.noreply.github.com> Co-authored-by: Daniel Tomes <10840635+GeekSheikh@users.noreply.github.com>

* Initial commit * autoOptimize enabled (#902) * autoOptimize enabled * review comments implemented * reordered function calls * updated func name and comments --------- Co-authored-by: Daniel Tomes <10840635+GeekSheikh@users.noreply.github.com> * Ambiguous API call error msg fixed (#944) * Initial commit * code fix --------- Co-authored-by: Guenia Izquierdo <guenia.izquierdo@databricks.com> * 906 update instance details (#983) * local in progress * finished and tested * added todo for 904 schema validation * activeUntil update logic fixed * added master schemas for instanceDetails and dbuCostDetails * updated for pr review * Add Verify Min Schema in Audit Log Bronze (#1009) * refactored database write function (#970) * Initial commit * refactored database.scala * reformatted func signature and renamed cache func * adding more case statement in write function * renamed the functions as per review comments --------- Co-authored-by: Guenia Izquierdo <guenia.izquierdo@databricks.com> Co-authored-by: Daniel Tomes <10840635+GeekSheikh@users.noreply.github.com> * 926 gcp cluster logs (#941) * Initial commit * adding code to hanlde gcp multiworkspace cluster logs * added review comments * added condition to check local deployment for gcp clusterlog fix * added code to check for remote workspace * reformat fetchClusterLogConfiguration signature --------- Co-authored-by: Guenia Izquierdo <guenia.izquierdo@databricks.com> * [FEAT] - ETL_DataPathPrefix_Migration (#918) * Initial commit * Snapshot Incremental * Snapshot Incremental * Snapshot Incremental * Snapshot Incremental * Verbose Audit Logging * Incremental Snapshot * Incremental Snapshot * ETL Migration * Snapshot Incremental * Snapshot Incremental * Snapshot Incremental * Snapshot Incremental * Added Restore Logic * Add remoteWorkspaceByPath in Migration script * Add Changes * Add Changes * Add Changes * Add Changes * Add Changes * Add Changes * Add Changes * Added changes as per comment from Guinea * Change as per latest PR comments * Change as per latest PR comments * Change as per latest PR comments * added support for delta table in updateConfig * Change the flow in Migration. Drop Database should be done after updateConfig * Change the flow in Migration. Drop Database should be done after updateConfig * Initial commit * autoOptimize enabled (#902) * autoOptimize enabled * review comments implemented * reordered function calls * updated func name and comments --------- Co-authored-by: Daniel Tomes <10840635+GeekSheikh@users.noreply.github.com> * Ambiguous API call error msg fixed (#944) * Initial commit * code fix --------- Co-authored-by: Guenia Izquierdo <guenia.izquierdo@databricks.com> * Snapshot Incremental * Snapshot Incremental * Snapshot Incremental * Snapshot Incremental * Verbose Audit Logging * Incremental Snapshot * Incremental Snapshot * ETL Migration * Snapshot Incremental * Snapshot Incremental * Snapshot Incremental * Snapshot Incremental * Added Restore Logic * Add remoteWorkspaceByPath in Migration script * Add Changes * Add Changes * Add Changes * Add Changes * Add Changes * Add Changes * Add Changes * Added changes as per comment from Guinea * Change as per latest PR comments * Change as per latest PR comments * Change as per latest PR comments * added support for delta table in updateConfig * Change the flow in Migration. Drop Database should be done after updateConfig * Change the flow in Migration. Drop Database should be done after updateConfig * Omit Main Method * Omit Main Method * Omit Main Method * Omit Main Method * Added Runner for both Migration.scala and Snapshot.scala * Added Runner for Restore.scala * Added Runner for Restore.scala * Added Runner for Restore.scala * Change the writeStream function Name * Added Validation for Snapshot.scala * Added Validation for Snapshot.scala * Added Validation for Snapshot.scala * Added Validation for Migration.scala and Restore.scala * Added Validation for Migration.scala and Restore.scala * Added Validation for Migration.scala and Restore.scala * Added Validation for Migration.scala and Restore.scala * Added Validation for Migration.scala and Restore.scala * Add Verify Min Schema in audit_log_bronze * Add Verify Min Schema in audit_log_bronze * Changed Apply function to Process * Change the Runner script to run as per the new process function * Change the Runner script to run as per the new process function * Change the Runner script to run as per the new process function * Change the Runner script to run as per the new process function * Removed println for Zone Validation * Merge 721 * Changed the negative condition in Snapshot.scala * Resolve the Conflict in Database.scala --------- Co-authored-by: Guenia Izquierdo <guenia.izquierdo@databricks.com> Co-authored-by: Sourav Banerjee <30810740+Sourav692@users.noreply.github.com> Co-authored-by: Sriram Mohanty <69749553+sriram251-code@users.noreply.github.com> Co-authored-by: Daniel Tomes <10840635+GeekSheikh@users.noreply.github.com> * 931 shuffle factor -- mega (#990) * Initial commit * autoOptimize enabled (#902) * autoOptimize enabled * review comments implemented * reordered function calls * updated func name and comments --------- Co-authored-by: Daniel Tomes <10840635+GeekSheikh@users.noreply.github.com> * Ambiguous API call error msg fixed (#944) * Initial commit * code fix --------- Co-authored-by: Guenia Izquierdo <guenia.izquierdo@databricks.com> * 931 shuffle factor -- mega --------- Co-authored-by: Guenia Izquierdo <guenia.izquierdo@databricks.com> Co-authored-by: Sriram Mohanty <69749553+sriram251-code@users.noreply.github.com> * code fix (#1022) * Update_Azure_gcp_instances (#982) * autoOptimize enabled (#902) * autoOptimize enabled * review comments implemented * reordered function calls * updated func name and comments --------- Co-authored-by: Daniel Tomes <10840635+GeekSheikh@users.noreply.github.com> * Update_Azure_gcp_instances Update_Azure_gcp_instances --------- Co-authored-by: Sriram Mohanty <69749553+sriram251-code@users.noreply.github.com> Co-authored-by: Daniel Tomes <10840635+GeekSheikh@users.noreply.github.com> * 462 dbsql warehouse (#674) * Added additional edge cases to StringExt and include White Spaces as allowable column special character * squash commits adding filter for response code 200 removed ambiguity from audit log schema adding status code 200 as filter for deriving warehouseId implemented review comments added dbsql etl columns in audit log schema adding dbsqlTransform and review comments changed module number for Bronze Warehouse snapshot changed the ETL definition for dbsql warehouse in all layers removed comments, changed key for warehouse bronze rearranging columns in gold and silver fixing silver and gold layer adding dbsql warehouse code * added review comments * adding code review comments * adding code review comments * code review comments * implementing code review comments * added dependency for silver warehouse and schema changes * removing commented code --------- Co-authored-by: SundarShankar89 <sundar.shankar@databricks.com> * Issue 928. Correct the cluster_type captured (#989) * resolving bug in StructToMap (#1016) * fix cluster log (#1028) * code fix * code fix * review comment implemented * review comment implemented * review comment implemented * Change Azure_instance_Detail.csv (#1027) Co-authored-by: Sourav Banerjee <30810740+Sourav692@users.noreply.github.com> * 743 rollback improvements (#986) * Initial commit * autoOptimize enabled (#902) * autoOptimize enabled * review comments implemented * reordered function calls * updated func name and comments --------- Co-authored-by: Daniel Tomes <10840635+GeekSheikh@users.noreply.github.com> * Ambiguous API call error msg fixed (#944) * Initial commit * code fix --------- Co-authored-by: Guenia Izquierdo <guenia.izquierdo@databricks.com> * 981 - Remove assertion and replace with warning * 743 - improved supported scenarios for rollbackPipelineForModule --------- Co-authored-by: Guenia Izquierdo <guenia.izquierdo@databricks.com> Co-authored-by: Sriram Mohanty <69749553+sriram251-code@users.noreply.github.com> * [FEAT] - Verbose audit logging 721 (#917) * Initial commit * Snapshot Incremental * Snapshot Incremental * Verbose Audit Logging * Verbose Audit Logging * Verbose Audit Logging * Add Changes * Add Changes * Add Changes * Add Changes * Add Changes * Add Changes * Initial commit * autoOptimize enabled (#902) * autoOptimize enabled * review comments implemented * reordered function calls * updated func name and comments --------- Co-authored-by: Daniel Tomes <10840635+GeekSheikh@users.noreply.github.com> * Ambiguous API call error msg fixed (#944) * Initial commit * code fix --------- Co-authored-by: Guenia Izquierdo <guenia.izquierdo@databricks.com> * Snapshot Incremental * Snapshot Incremental * Verbose Audit Logging * Verbose Audit Logging * Verbose Audit Logging * Add Changes * Add Changes * Add Changes * Add Changes * Add Changes * Add Changes * add as per comment * Change CLSF impute logic * Change CLSF impute logic * Resolve the Conflict in Database.scala * Resolve the Conflict in Database.scala * Resolve the Conflict in Database.scala * Resolve the Conflict in Database.scala * Resolve the Conflict in Database.scala * Change Azure_instance_Detail.csv * Change Azure_instance_Detail.csv * Change Azure_instance_Detail.csv * Change Azure_instance_Detail.csv * Change Window function to GroupBy in buildNotebookCommandsFact * Rollback the groupBy again to window function. * Change the window functions name in buildNotebookCommandsFact --------- Co-authored-by: Guenia Izquierdo <guenia.izquierdo@databricks.com> Co-authored-by: Sourav Banerjee <30810740+Sourav692@users.noreply.github.com> Co-authored-by: Sriram Mohanty <69749553+sriram251-code@users.noreply.github.com> Co-authored-by: Daniel Tomes <10840635+GeekSheikh@users.noreply.github.com> * Issue 957. Rectified the resize Logic. (#988) * adding code to handle null response from cluster policy api (#1032) * notebook_commands is failing when verbose Audit Logging is not enabled in workspace (#1034) * Hot fix for notebookCommands_gold when verbose audit logging is not enabled in workspace * Change the error logging message for notebookCommands_gold * Change the error logging message for notebookCommands_gold * Change the error logging message for notebookCommands_gold-Version 2 * Change the error logging message for notebookCommands_gold-Version 3 * Adding scope for notebook commands * Update InitializerFunctions.scala Added NotebookCommands in InitializerFunctions.scala * Update GoldTransforms.scala Change the Error Msg in buildNotebookCommandsFact * Added NotebookCommands in InitializerFunctions.scala * Add if-else in notebooksCommands * Added NotebookCommands in InitializerFunctions.scala * Added NotebookCommands scope in MultiWorkspaceDeployment.scala * Change as per latest comment --------- Co-authored-by: Sourav Banerjee <30810740+Sourav692@users.noreply.github.com> Co-authored-by: Guenia Izquierdo <guenia.izquierdo@databricks.com> --------- Co-authored-by: Sriram Mohanty <69749553+sriram251-code@users.noreply.github.com> Co-authored-by: Daniel Tomes <10840635+GeekSheikh@users.noreply.github.com> Co-authored-by: Sourav Banerjee <109206082+souravbaner-da@users.noreply.github.com> Co-authored-by: Aman <91308367+aman-db@users.noreply.github.com> Co-authored-by: Sourav Banerjee <30810740+Sourav692@users.noreply.github.com> Co-authored-by: mohanbaabu1996 <87074323+mohanbaabu1996@users.noreply.github.com> Co-authored-by: SundarShankar89 <sundar.shankar@databricks.com> Co-authored-by: brij-raghuwanshi-db <brijendra.raghuwanshi@databricks.com>

sriram251-code added the feature New feature or extension of existing feature (was "enhancement") label Apr 24, 2023

sriram251-code added this to the 0.7.3.0 milestone Apr 24, 2023

sriram251-code requested review from GeekSheikh and gueniai April 24, 2023 14:40

sriram251-code self-assigned this Apr 24, 2023

gueniai modified the milestones: 0.7.3.0, 0.7.2.1 Apr 25, 2023

sriram251-code requested review from aman-db and souravbaner-da May 2, 2023 13:12

aman-db approved these changes May 3, 2023

View reviewed changes

GeekSheikh suggested changes May 5, 2023

View reviewed changes

sriram251-code requested a review from GeekSheikh May 8, 2023 09:07

GeekSheikh linked an issue May 15, 2023 that may be closed by this pull request

[BUG] - AutoOptimize not getting set #850

Closed

GeekSheikh changed the base branch from 0730_release to 0721_release May 15, 2023 13:25

GeekSheikh suggested changes May 18, 2023

View reviewed changes

sriram251-code requested a review from GeekSheikh May 19, 2023 10:39

sriram251-code requested a review from aman-db May 22, 2023 13:34

GeekSheikh force-pushed the 0721_release branch from 36a8591 to 6c3a11b Compare June 23, 2023 12:10

sriram251-code and others added 3 commits June 27, 2023 13:29

autoOptimize enabled

60b1d9f

review comments implemented

ec3d19f

reordered function calls

e43481e

GeekSheikh force-pushed the 850_autoOptimize_730 branch from 8e3178b to e43481e Compare June 27, 2023 17:29

updated func name and comments

874f981

GeekSheikh approved these changes Jun 27, 2023

View reviewed changes

GeekSheikh merged commit e4ea15c into 0721_release Jun 27, 2023

GeekSheikh deleted the 850_autoOptimize_730 branch June 27, 2023 17:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

autoOptimize enabled #902

autoOptimize enabled #902

sriram251-code commented Apr 24, 2023 •

edited

Loading

GeekSheikh commented Apr 28, 2023

GeekSheikh left a comment

GeekSheikh May 5, 2023

GeekSheikh May 5, 2023

sriram251-code May 8, 2023

GeekSheikh May 5, 2023

sriram251-code May 8, 2023

GeekSheikh May 5, 2023

sriram251-code May 8, 2023

GeekSheikh May 5, 2023

sriram251-code May 8, 2023

GeekSheikh May 5, 2023

sriram251-code May 8, 2023

GeekSheikh left a comment

GeekSheikh May 18, 2023

GeekSheikh May 18, 2023

GeekSheikh May 18, 2023

GeekSheikh May 18, 2023

GeekSheikh May 18, 2023

GeekSheikh May 18, 2023

GeekSheikh May 18, 2023

GeekSheikh May 18, 2023

GeekSheikh May 18, 2023

GeekSheikh May 18, 2023

sonarcloud bot commented May 22, 2023

sonarcloud bot commented Jun 27, 2023

		if (!targetPropertiesAccurate) {
		val alterStatement = s"""alter table ${target.tableFullName} set TBLPROPERTIES (delta.autoOptimize.optimizeWrite = true)"""

autoOptimize enabled #902

autoOptimize enabled #902

Conversation

sriram251-code commented Apr 24, 2023 • edited Loading

GeekSheikh commented Apr 28, 2023

GeekSheikh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GeekSheikh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarcloud bot commented May 22, 2023

sonarcloud bot commented Jun 27, 2023

sriram251-code commented Apr 24, 2023 •

edited

Loading