Listing improvements: new directory structure for array. (#2918)

* Listing improvements: new directory structure for array. When opening an array, certain remote file systems limit the number of elements returned (for example: 1000 for s3). For arrays with a large number of fragments can take a long time. In an effort to reduce the size of the listing operations, this PR shuffles a few elements of the array directory. First of all, it moves all the ok files to a __commit folder and the fragments to a __fragment folder. Second, it moves the consolidated fragment metadata to a __fragment_meta directory. Finally, this PR also ensures that arrays with the old format, the new format or a mix of both can be read. --- TYPE: IMPROVEMENT DESC: Listing improvements: new directory structure for array. * Addressing feedback from @stavrospapadopoulos.
TileDB-Inc · Feb 27, 2022 · ce204ad · ce204ad
1 parent 36fdf76
commit ce204ad
Show file tree

Hide file tree

Showing 26 changed files with 527 additions and 278 deletions.
diff --git a/format_spec/FORMAT_SPEC.md b/format_spec/FORMAT_SPEC.md
@@ -2,7 +2,7 @@
 
 
 :information_source: **Notes:**  
-- The current TileDB format version number is **11** (`uint32_t`).
+- The current TileDB format version number is **12** (`uint32_t`).
 - All data written by TileDB and referenced in this document is **little-endian**. 
 
 ## Table of Contents

diff --git a/format_spec/array_file_hierarchy.md b/format_spec/array_file_hierarchy.md
@@ -3,16 +3,19 @@
 An array is a folder with the following structure:
 
 ```
-my_array                          # array folder
-    |_ __schema                   # array schema folder
-    |_ <timestamped_name>         # fragment folder
-    |_ <timestamped_name>.ok      # fragment ok file
-    |_ ...
-    |_ <timestamped_name>.vac     # fragment vacuum file
-    |_ ...                  
-    |_ <timestamped_name>.meta    # consol. fragment meta file
-    |_ ...                  
-    |_ __meta                     # array metadata folder
+my_array                                # array folder
+    |_ __schema                         # array schema folder
+    |_ __fragments                      # array fragments folder
+          |_ <timestamped_name>         # fragment folder
+          |_ ...
+    |_ __commits                        # array commits folder
+          |_ <timestamped_name>.wrt     # fragment write file
+          |_ ...
+          |_ <timestamped_name>.vac     # fragment vacuum file
+    |_ __fragment_meta                  
+          |_ <timestamped_name>.meta    # consol. fragment meta file
+          |_ ...                  
+    |_ __meta                           # array metadata folder
 
 ```
 
@@ -24,8 +27,8 @@ A `<timestamped_name>` above has format `__t1_t2_uuid_v`, where
 Inside the array folder, you can find the following:
 
 * [Array schema](./array_schema.md) folder `__schema`.
-* Any number of [fragment folders](./fragment.md) `<timestamped_name>`.
-* An empty file `<timestamped_name>.ok` associated with every fragment folder `<timestamped_name>`, where `<timestamped_name>` is common for the folder and the OK file. This is used to indicate that fragment `<timestamped_name>` has been *committed* (i.e., its write process finished successfully) and it is ready for use by TileDB. If the OK file does not exist, the corresponding fragment folder is ignored by TileDB during the reads.
-* Any number of [vacuum files](./vacuum_file.md) of the form `<timestamped_name>.vac`.
-* Any number of [consolidated fragment metadata files](./consolidated_fragment_metadata_file.md) of the form `<timestamped_name>.meta`.
+* Inside of a fragments folder, any number of [fragment folders](./fragment.md) `<timestamped_name>`.
+* Inside of a commit folder, an empty file `<timestamped_name>.wrt` associated with every fragment folder `<timestamped_name>`, where `<timestamped_name>` is common for the folder and the WRT file. This is used to indicate that fragment `<timestamped_name>` has been *committed* (i.e., its write process finished successfully) and it is ready for use by TileDB. If the WRT file does not exist, the corresponding fragment folder is ignored by TileDB during the reads.
+* Inside the same commit folder, any number of [vacuum files](./vacuum_file.md) of the form `<timestamped_name>.vac`.
+* Inside of a fragment metadata folder, any number of [consolidated fragment metadata files](./consolidated_fragment_metadata_file.md) of the form `<timestamped_name>.meta`.
 * [Array metadata](./array_metadata.md) folder `__meta`.
diff --git a/format_spec/consolidated_fragment_metadata_file.md b/format_spec/consolidated_fragment_metadata_file.md
@@ -3,10 +3,11 @@
 A consolidated fragment metadata file has name `<timestamped_name>.meta` and is located here:
 
 ```
-my_array                        # array folder
+my_array                              # array folder
    |_ ....
-   |_ <timestamped_name>.meta   # consolidated fragment metadata file
-   |_ ...
+   |_ __fragment_meta                 # array fragment metadata folder
+         |_ <timestamped_name>.meta   # consolidated fragment metadata file
+         |_ ...
 ```
 
 `<timestamped_name>` has format `__t1_t2_uuid_v`, where:

diff --git a/format_spec/fragment.md b/format_spec/fragment.md
@@ -5,19 +5,20 @@
 A fragment metadata folder is called `<timestamped_name>` and located here:
 
 ```
-my_array                              # array folder
+my_array                                    # array folder
    |  ...
-   |_ <timestamped_name>              # fragment folder
-   |      |_ __fragment_metadata.tdb  # fragment metadata
-   |      |_ a0.tdb                   # fixed-sized attribute 
-   |      |_ a1.tdb                   # var-sized attribute (offsets) 
-   |      |_ a1_var.tdb               # var-sized attribute (values)
-   |      |_ ...      
-   |      |_ d0.tdb                   # fixed-sized dimension 
-   |      |_ d1.tdb                   # var-sized dimension (offsets) 
-   |      |_ d1_var.tdb               # var-sized dimension (values)
-   |      |_ ...      
-   |_ ...  
+   |_ __fragments                           # array fragments folder
+         |_ <timestamped_name>              # fragment folder
+         |      |_ __fragment_metadata.tdb  # fragment metadata
+         |      |_ a0.tdb                   # fixed-sized attribute 
+         |      |_ a1.tdb                   # var-sized attribute (offsets) 
+         |      |_ a1_var.tdb               # var-sized attribute (values)
+         |      |_ ...      
+         |      |_ d0.tdb                   # fixed-sized dimension 
+         |      |_ d1.tdb                   # var-sized dimension (offsets) 
+         |      |_ d1_var.tdb               # var-sized dimension (values)
+         |      |_ ...      
+        |_ ...  
 ```
 
 `<timestamped_name>` has format `__t1_t2_uuid_v`, where:

diff --git a/format_spec/vacuum_file.md b/format_spec/vacuum_file.md
@@ -1,12 +1,12 @@
 # Vacuum File
 
-A vacuum file has name `__t1_t2_uuid_v.vac` and can be located either in the array folder:
+A vacuum file has name `__t1_t2_uuid_v.vac` and can be located either in the array commit folder:
 
 ```
 my_array                        # array folder
    |_ ....
-   |_ __t1_t2_uuid_v.vac        # vacuum file
-   |_ ...
+   |_ __commits                 # array commit folder
+         |___t1_t2_uuid_v.vac   # vacuum file
 ```
 
 or in the array metadata folder:

diff --git a/test/src/helpers.cc b/test/src/helpers.cc
@@ -1221,19 +1221,9 @@ int32_t num_fragments(const std::string& array_name) {
   VFS vfs(ctx);
 
   // Get all URIs in the array directory
-  auto uris = vfs.ls(array_name);
-
-  // Exclude '__meta' directory and any file with a suffix
-  int ret = 0;
-  for (const auto& uri : uris) {
-    auto name = tiledb::sm::URI(uri).remove_trailing_slash().last_path_part();
-    if (name != tiledb::sm::constants::array_metadata_dir_name &&
-        name != tiledb::sm::constants::array_schema_dir_name &&
-        name.find_first_of('.') == std::string::npos)
-      ++ret;
-  }
-
-  return ret;
+  auto uris = vfs.ls(
+      array_name + "/" + tiledb::sm::constants::array_fragments_dir_name);
+  return static_cast<uint32_t>(uris.size());
 }
 
 std::string random_string(const uint64_t l) {
@@ -1251,6 +1241,14 @@ std::string random_string(const uint64_t l) {
   return s;
 }
 
+std::string get_fragment_dir(std::string array_dir) {
+  return array_dir + "/" + tiledb::sm::constants::array_fragments_dir_name;
+}
+
+std::string get_commit_dir(std::string array_dir) {
+  return array_dir + "/" + tiledb::sm::constants::array_commit_dir_name;
+}
+
 template void check_subarray<int8_t>(
     tiledb::sm::Subarray& subarray, const SubarrayRanges<int8_t>& ranges);
 template void check_subarray<uint8_t>(

diff --git a/test/src/helpers.h b/test/src/helpers.h
@@ -681,6 +681,16 @@ int32_t num_fragments(const std::string& array_name);
  */
 std::string random_string(const uint64_t l);
 
+/**
+ * Gets the fragments directory from the array directory.
+ */
+std::string get_fragment_dir(std::string array_dir);
+
+/**
+ * Gets the commit directory from the array directory.
+ */
+std::string get_commit_dir(std::string array_dir);
+
 }  // End of namespace test
 
 }  // End of namespace tiledb

diff --git a/test/src/unit-ArrayDirectory.cc b/test/src/unit-ArrayDirectory.cc
@@ -5,7 +5,7 @@
  *
  * The MIT License
  *
- * @copyright Copyright (c) 2017-2021 TileDB, Inc.
+ * @copyright Copyright (c) 2017-2022 TileDB, Inc.
  *
  * Permission is hereby granted, free of charge, to any person obtaining a copy
  * of this software and associated documentation files (the "Software"), to deal

diff --git a/test/src/unit-backwards_compat.cc b/test/src/unit-backwards_compat.cc
@@ -32,6 +32,7 @@
  */
 
 #include "catch.hpp"
+#include "test/src/helpers.h"
 #include "tiledb/common/common.h"
 #include "tiledb/sm/cpp_api/tiledb"
 #include "tiledb/sm/misc/constants.h"
@@ -42,6 +43,7 @@
 #include <thread>
 
 using namespace tiledb;
+using namespace tiledb::test;
 
 namespace {
 
@@ -694,7 +696,6 @@ TEST_CASE(
         .set_data_buffer("a", a_write)
         .set_coordinates(coords_write);
     query_w.submit();
-    fragment_uri = query_w.fragment_uri(0);
     old_array.close();
 
     // Read
@@ -713,8 +714,8 @@ TEST_CASE(
 
     // Remove created fragment and ok file
     VFS vfs(ctx);
-    vfs.remove_dir(fragment_uri);
-    vfs.remove_file(fragment_uri + ".ok");
+    vfs.remove_dir(get_fragment_dir(old_array_name));
+    vfs.remove_dir(get_commit_dir(old_array_name));
 
     REQUIRE(a_read[0] == 100);
     for (int i = 1; i < 4; i++) {
@@ -1306,7 +1307,7 @@ TEST_CASE(
   schema_folder = array_read2.uri() + "/__schema";
 
   VFS vfs(ctx);
-  vfs.remove_dir(fragment_uri);
-  vfs.remove_file(fragment_uri + ".ok");
+  vfs.remove_dir(get_fragment_dir(array_read2.uri()));
+  vfs.remove_dir(get_commit_dir(array_read2.uri()));
   vfs.remove_dir(schema_folder);
 }
diff --git a/test/src/unit-capi-array.cc b/test/src/unit-capi-array.cc
@@ -130,7 +130,7 @@ int ArrayFx::get_fragment_timestamps(const char* path, void* data) {
   auto data_vec = (std::vector<uint64_t>*)data;
   std::pair<uint64_t, uint64_t> timestamp_range;
   if (tiledb::sm::utils::parse::ends_with(
-          path, tiledb::sm::constants::ok_file_suffix)) {
+          path, tiledb::sm::constants::write_file_suffix)) {
     auto uri = tiledb::sm::URI(path);
     if (tiledb::sm::utils::parse::get_timestamp_range(uri, &timestamp_range)
             .ok())
@@ -985,7 +985,7 @@ TEST_CASE_METHOD(
   rc = tiledb_vfs_ls(
       ctx_,
       vfs_,
-      array_name.c_str(),
+      get_commit_dir(array_name).c_str(),
       &get_fragment_timestamps,
       &fragment_timestamps);
   CHECK(rc == TILEDB_OK);