926 gcp cluster logs #941
Merged
926 gcp cluster logs #941
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
aman-db
added
feature
|
Kudos, SonarCloud Quality Gate passed!
|
aman-db
force-pushed
the
926_gcp_cluster_logs
branch
from
09a5263
to
b258601
Compare
GeekSheikh
previously requested changes
Jun 27, 2023
src/main/scala/com/databricks/labs/overwatch/pipeline/BronzeTransforms.scala
Outdated
Show resolved
Hide resolved
src/main/scala/com/databricks/labs/overwatch/pipeline/BronzeTransforms.scala
Outdated
Show resolved
Hide resolved
src/main/scala/com/databricks/labs/overwatch/pipeline/BronzeTransforms.scala
Outdated
Show resolved
Hide resolved
sriram251-code
requested changes
Jun 28, 2023
src/main/scala/com/databricks/labs/overwatch/pipeline/BronzeTransforms.scala
Outdated
Show resolved
Hide resolved
sriram251-code
requested changes
Jul 11, 2023
src/main/scala/com/databricks/labs/overwatch/pipeline/BronzeTransforms.scala
Outdated
Show resolved
Hide resolved
|
Kudos, SonarCloud Quality Gate passed!
|
sriram251-code
requested changes
Jul 19, 2023
@sriram251-code Done. Please check |
sriram251-code
approved these changes
Aug 3, 2023
gueniai
added a commit
that referenced
this pull request
Sep 11, 2023
* Initial commit * autoOptimize enabled (#902) * autoOptimize enabled * review comments implemented * reordered function calls * updated func name and comments --------- Co-authored-by: Daniel Tomes <10840635+GeekSheikh@users.noreply.github.com> * Ambiguous API call error msg fixed (#944) * Initial commit * code fix --------- Co-authored-by: Guenia Izquierdo <guenia.izquierdo@databricks.com> * 906 update instance details (#983) * local in progress * finished and tested * added todo for 904 schema validation * activeUntil update logic fixed * added master schemas for instanceDetails and dbuCostDetails * updated for pr review * Add Verify Min Schema in Audit Log Bronze (#1009) * refactored database write function (#970) * Initial commit * refactored database.scala * reformatted func signature and renamed cache func * adding more case statement in write function * renamed the functions as per review comments --------- Co-authored-by: Guenia Izquierdo <guenia.izquierdo@databricks.com> Co-authored-by: Daniel Tomes <10840635+GeekSheikh@users.noreply.github.com> * 926 gcp cluster logs (#941) * Initial commit * adding code to hanlde gcp multiworkspace cluster logs * added review comments * added condition to check local deployment for gcp clusterlog fix * added code to check for remote workspace * reformat fetchClusterLogConfiguration signature --------- Co-authored-by: Guenia Izquierdo <guenia.izquierdo@databricks.com> * [FEAT] - ETL_DataPathPrefix_Migration (#918) * Initial commit * Snapshot Incremental * Snapshot Incremental * Snapshot Incremental * Snapshot Incremental * Verbose Audit Logging * Incremental Snapshot * Incremental Snapshot * ETL Migration * Snapshot Incremental * Snapshot Incremental * Snapshot Incremental * Snapshot Incremental * Added Restore Logic * Add remoteWorkspaceByPath in Migration script * Add Changes * Add Changes * Add Changes * Add Changes * Add Changes * Add Changes * Add Changes * Added changes as per comment from Guinea * Change as per latest PR comments * Change as per latest PR comments * Change as per latest PR comments * added support for delta table in updateConfig * Change the flow in Migration. Drop Database should be done after updateConfig * Change the flow in Migration. Drop Database should be done after updateConfig * Initial commit * autoOptimize enabled (#902) * autoOptimize enabled * review comments implemented * reordered function calls * updated func name and comments --------- Co-authored-by: Daniel Tomes <10840635+GeekSheikh@users.noreply.github.com> * Ambiguous API call error msg fixed (#944) * Initial commit * code fix --------- Co-authored-by: Guenia Izquierdo <guenia.izquierdo@databricks.com> * Snapshot Incremental * Snapshot Incremental * Snapshot Incremental * Snapshot Incremental * Verbose Audit Logging * Incremental Snapshot * Incremental Snapshot * ETL Migration * Snapshot Incremental * Snapshot Incremental * Snapshot Incremental * Snapshot Incremental * Added Restore Logic * Add remoteWorkspaceByPath in Migration script * Add Changes * Add Changes * Add Changes * Add Changes * Add Changes * Add Changes * Add Changes * Added changes as per comment from Guinea * Change as per latest PR comments * Change as per latest PR comments * Change as per latest PR comments * added support for delta table in updateConfig * Change the flow in Migration. Drop Database should be done after updateConfig * Change the flow in Migration. Drop Database should be done after updateConfig * Omit Main Method * Omit Main Method * Omit Main Method * Omit Main Method * Added Runner for both Migration.scala and Snapshot.scala * Added Runner for Restore.scala * Added Runner for Restore.scala * Added Runner for Restore.scala * Change the writeStream function Name * Added Validation for Snapshot.scala * Added Validation for Snapshot.scala * Added Validation for Snapshot.scala * Added Validation for Migration.scala and Restore.scala * Added Validation for Migration.scala and Restore.scala * Added Validation for Migration.scala and Restore.scala * Added Validation for Migration.scala and Restore.scala * Added Validation for Migration.scala and Restore.scala * Add Verify Min Schema in audit_log_bronze * Add Verify Min Schema in audit_log_bronze * Changed Apply function to Process * Change the Runner script to run as per the new process function * Change the Runner script to run as per the new process function * Change the Runner script to run as per the new process function * Change the Runner script to run as per the new process function * Removed println for Zone Validation * Merge 721 * Changed the negative condition in Snapshot.scala * Resolve the Conflict in Database.scala --------- Co-authored-by: Guenia Izquierdo <guenia.izquierdo@databricks.com> Co-authored-by: Sourav Banerjee <30810740+Sourav692@users.noreply.github.com> Co-authored-by: Sriram Mohanty <69749553+sriram251-code@users.noreply.github.com> Co-authored-by: Daniel Tomes <10840635+GeekSheikh@users.noreply.github.com> * 931 shuffle factor -- mega (#990) * Initial commit * autoOptimize enabled (#902) * autoOptimize enabled * review comments implemented * reordered function calls * updated func name and comments --------- Co-authored-by: Daniel Tomes <10840635+GeekSheikh@users.noreply.github.com> * Ambiguous API call error msg fixed (#944) * Initial commit * code fix --------- Co-authored-by: Guenia Izquierdo <guenia.izquierdo@databricks.com> * 931 shuffle factor -- mega --------- Co-authored-by: Guenia Izquierdo <guenia.izquierdo@databricks.com> Co-authored-by: Sriram Mohanty <69749553+sriram251-code@users.noreply.github.com> * code fix (#1022) * Update_Azure_gcp_instances (#982) * autoOptimize enabled (#902) * autoOptimize enabled * review comments implemented * reordered function calls * updated func name and comments --------- Co-authored-by: Daniel Tomes <10840635+GeekSheikh@users.noreply.github.com> * Update_Azure_gcp_instances Update_Azure_gcp_instances --------- Co-authored-by: Sriram Mohanty <69749553+sriram251-code@users.noreply.github.com> Co-authored-by: Daniel Tomes <10840635+GeekSheikh@users.noreply.github.com> * 462 dbsql warehouse (#674) * Added additional edge cases to StringExt and include White Spaces as allowable column special character * squash commits adding filter for response code 200 removed ambiguity from audit log schema adding status code 200 as filter for deriving warehouseId implemented review comments added dbsql etl columns in audit log schema adding dbsqlTransform and review comments changed module number for Bronze Warehouse snapshot changed the ETL definition for dbsql warehouse in all layers removed comments, changed key for warehouse bronze rearranging columns in gold and silver fixing silver and gold layer adding dbsql warehouse code * added review comments * adding code review comments * adding code review comments * code review comments * implementing code review comments * added dependency for silver warehouse and schema changes * removing commented code --------- Co-authored-by: SundarShankar89 <sundar.shankar@databricks.com> * Issue 928. Correct the cluster_type captured (#989) * resolving bug in StructToMap (#1016) * fix cluster log (#1028) * code fix * code fix * review comment implemented * review comment implemented * review comment implemented * Change Azure_instance_Detail.csv (#1027) Co-authored-by: Sourav Banerjee <30810740+Sourav692@users.noreply.github.com> * 743 rollback improvements (#986) * Initial commit * autoOptimize enabled (#902) * autoOptimize enabled * review comments implemented * reordered function calls * updated func name and comments --------- Co-authored-by: Daniel Tomes <10840635+GeekSheikh@users.noreply.github.com> * Ambiguous API call error msg fixed (#944) * Initial commit * code fix --------- Co-authored-by: Guenia Izquierdo <guenia.izquierdo@databricks.com> * 981 - Remove assertion and replace with warning * 743 - improved supported scenarios for rollbackPipelineForModule --------- Co-authored-by: Guenia Izquierdo <guenia.izquierdo@databricks.com> Co-authored-by: Sriram Mohanty <69749553+sriram251-code@users.noreply.github.com> * [FEAT] - Verbose audit logging 721 (#917) * Initial commit * Snapshot Incremental * Snapshot Incremental * Verbose Audit Logging * Verbose Audit Logging * Verbose Audit Logging * Add Changes * Add Changes * Add Changes * Add Changes * Add Changes * Add Changes * Initial commit * autoOptimize enabled (#902) * autoOptimize enabled * review comments implemented * reordered function calls * updated func name and comments --------- Co-authored-by: Daniel Tomes <10840635+GeekSheikh@users.noreply.github.com> * Ambiguous API call error msg fixed (#944) * Initial commit * code fix --------- Co-authored-by: Guenia Izquierdo <guenia.izquierdo@databricks.com> * Snapshot Incremental * Snapshot Incremental * Verbose Audit Logging * Verbose Audit Logging * Verbose Audit Logging * Add Changes * Add Changes * Add Changes * Add Changes * Add Changes * Add Changes * add as per comment * Change CLSF impute logic * Change CLSF impute logic * Resolve the Conflict in Database.scala * Resolve the Conflict in Database.scala * Resolve the Conflict in Database.scala * Resolve the Conflict in Database.scala * Resolve the Conflict in Database.scala * Change Azure_instance_Detail.csv * Change Azure_instance_Detail.csv * Change Azure_instance_Detail.csv * Change Azure_instance_Detail.csv * Change Window function to GroupBy in buildNotebookCommandsFact * Rollback the groupBy again to window function. * Change the window functions name in buildNotebookCommandsFact --------- Co-authored-by: Guenia Izquierdo <guenia.izquierdo@databricks.com> Co-authored-by: Sourav Banerjee <30810740+Sourav692@users.noreply.github.com> Co-authored-by: Sriram Mohanty <69749553+sriram251-code@users.noreply.github.com> Co-authored-by: Daniel Tomes <10840635+GeekSheikh@users.noreply.github.com> * Issue 957. Rectified the resize Logic. (#988) * adding code to handle null response from cluster policy api (#1032) * notebook_commands is failing when verbose Audit Logging is not enabled in workspace (#1034) * Hot fix for notebookCommands_gold when verbose audit logging is not enabled in workspace * Change the error logging message for notebookCommands_gold * Change the error logging message for notebookCommands_gold * Change the error logging message for notebookCommands_gold-Version 2 * Change the error logging message for notebookCommands_gold-Version 3 * Adding scope for notebook commands * Update InitializerFunctions.scala Added NotebookCommands in InitializerFunctions.scala * Update GoldTransforms.scala Change the Error Msg in buildNotebookCommandsFact * Added NotebookCommands in InitializerFunctions.scala * Add if-else in notebooksCommands * Added NotebookCommands in InitializerFunctions.scala * Added NotebookCommands scope in MultiWorkspaceDeployment.scala * Change as per latest comment --------- Co-authored-by: Sourav Banerjee <30810740+Sourav692@users.noreply.github.com> Co-authored-by: Guenia Izquierdo <guenia.izquierdo@databricks.com> --------- Co-authored-by: Sriram Mohanty <69749553+sriram251-code@users.noreply.github.com> Co-authored-by: Daniel Tomes <10840635+GeekSheikh@users.noreply.github.com> Co-authored-by: Sourav Banerjee <109206082+souravbaner-da@users.noreply.github.com> Co-authored-by: Aman <91308367+aman-db@users.noreply.github.com> Co-authored-by: Sourav Banerjee <30810740+Sourav692@users.noreply.github.com> Co-authored-by: mohanbaabu1996 <87074323+mohanbaabu1996@users.noreply.github.com> Co-authored-by: SundarShankar89 <sundar.shankar@databricks.com> Co-authored-by: brij-raghuwanshi-db <brijendra.raghuwanshi@databricks.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
closes #926