New Domain Metadata and "Regular" Metadata

japila-books · Feb 11, 2024 · a508180 · a508180
1 parent ba48bbf
commit a508180
Show file tree

Hide file tree

Showing 6 changed files with 48 additions and 5 deletions.
diff --git a/docs/DeltaLog.md b/docs/DeltaLog.md
@@ -94,15 +94,17 @@ val expected = new Path(s"file:$dataPath/_delta_log/_last_checkpoint")
 assert(deltaLog.LAST_CHECKPOINT == expected)
 ```
 
-## <span id="tableExists"> tableExists
+## tableExists { #tableExists }
 
 ```scala
 tableExists: Boolean
 ```
 
 `tableExists` requests the [current Snapshot](SnapshotManagement.md#snapshot) for the [version](Snapshot.md#version) and checks out whether it is `0` or higher.
 
-is used when:
+---
+
+`tableExists` is used when:
 
 * `DeltaTable` utility is used to [isDeltaTable](DeltaTable.md#isDeltaTable)
 * [DeltaUnsupportedOperationsCheck](DeltaUnsupportedOperationsCheck.md) logical check rule is executed

diff --git a/docs/DomainMetadataUtils.md b/docs/DomainMetadataUtils.md
@@ -0,0 +1,3 @@
+# DomainMetadataUtils
+
+`DomainMetadataUtils`...FIXME
diff --git a/docs/ImplicitMetadataOperation.md b/docs/ImplicitMetadataOperation.md
@@ -147,7 +147,7 @@ mergeSchema(
 
 `mergeSchema`...FIXME
 
-## New Domain Metadata { #getNewDomainMetadata }
+## New DomainMetadatas { #getNewDomainMetadata }
 
 ```scala
 getNewDomainMetadata(
@@ -157,7 +157,21 @@ getNewDomainMetadata(
   clusterBySpecOpt: Option[ClusterBySpec] = None): Seq[DomainMetadata]
 ```
 
-`getNewDomainMetadata`...FIXME
+`getNewDomainMetadata` is empty (no [DomainMetadata](DomainMetadata.md)) if either of the following holds:
+
+* The given `canUpdateMetadata` flag is `false`
+* The given `isReplacingTable` flag is `false` and the delta table (of the given [OptimisticTransaction](OptimisticTransaction.md#deltaLog)) [exists](DeltaLog.md#tableExists)
+
+??? note "`canUpdateMetadata` flag"
+    The input `canUpdateMetadata` flag is exactly [canUpdateMetadata](OptimisticTransactionImpl.md#canUpdateMetadata) of the given [OptimisticTransaction](OptimisticTransaction.md).
+
+??? note "`isReplacingTable` flag"
+    The input `isReplacingTable` flag holds true for the [SaveMode](commands/WriteIntoDelta.md#mode) being [Overwrite](commands/WriteIntoDelta.md#isOverwriteOperation) with no [replaceWhere](spark-connector/options.md#replaceWhere) option enabled.
+
+For all other cases, `getNewDomainMetadata` does one of the following:
+
+1. When the delta table (of the given [OptimisticTransaction](OptimisticTransaction.md#deltaLog)) does not [exist](DeltaLog.md#tableExists), `getNewDomainMetadata` gives a [DomainMetadata](liquid-clustering/ClusteredTableUtilsBase.md#getDomainMetadataOptional) for the given [ClusterBySpec](liquid-clustering/ClusterBySpec.md)
+1. Otherwise, `getNewDomainMetadata` [handles domain metadata for replacing a table](DomainMetadataUtils.md#handleDomainMetadataForReplaceTable) (with the [existing](SnapshotStateManager.md#domainMetadata) and the new clustered `DomainMetadata`)
 
 ---
 

diff --git a/docs/OptimisticTransactionImpl.md b/docs/OptimisticTransactionImpl.md
@@ -813,6 +813,25 @@ commitLarge(
 * [CreateDeltaTableCommand](commands/create-table/CreateDeltaTableCommand.md) (and [handleClone](commands/clone/CloneTableBase.md#handleClone))
 * [RESTORE](commands/restore/index.md)
 
+## Metadata Can Be Updated { #canUpdateMetadata }
+
+```scala
+canUpdateMetadata: Boolean
+```
+
+`canUpdateMetadata` holds when neither this transaction has already written data nor the metadata has been already changed.
+
+In other words, `canUpdateMetadata` holds `true` when both of the following hold:
+
+1. [hasWritten](TransactionalWrite.md#hasWritten) is `false`
+1. [newMetadata](#newMetadata) is undefined
+
+---
+
+`canUpdateMetadata` is used when:
+
+* `WriteIntoDelta` is requested to [write data out](commands/WriteIntoDelta.md#write)
+
 ## Logging
 
 `OptimisticTransactionImpl` is a Scala trait and logging is configured using the logger of the [implementations](#implementations).
diff --git a/docs/SnapshotStateManager.md b/docs/SnapshotStateManager.md
@@ -0,0 +1,3 @@
+# SnapshotStateManager
+
+`SnapshotStateManager`...FIXME
diff --git a/mkdocs.yml b/mkdocs.yml
@@ -202,9 +202,10 @@ nav:
     - Transaction Log (DeltaLog):
       - DeltaLog: DeltaLog.md
       - Operation: Operation.md
+      - ReadChecksum: ReadChecksum.md
       - SnapshotManagement: SnapshotManagement.md
       - SnapshotDescriptor.md
-      - ReadChecksum: ReadChecksum.md
+      - SnapshotStateManager: SnapshotStateManager.md
       - VerifyChecksum: VerifyChecksum.md
     - Optimistic Transactions:
       - OptimisticTransaction: OptimisticTransaction.md
@@ -333,6 +334,7 @@ nav:
     - DeltaRelation: DeltaRelation.md
     - DeltaTableOperations: DeltaTableOperations.md
     - DeltaTableUtils: DeltaTableUtils.md
+    - DomainMetadataUtils: DomainMetadataUtils.md
     - ImplicitMetadataOperation: ImplicitMetadataOperation.md
     - SchemaMergingUtils: SchemaMergingUtils.md
     - SchemaUtils: SchemaUtils.md