add option to also transform matset_values to silo (#685)

* add option to also transform matset_values to silo

* add venn mat_check field, change matset index to use material_map, fix bugs with relay mesh bp truncate write

CHANGELOG.md

@@ -31,6 +31,14 @@ and this project aspires to adhere to [Semantic Versioning](https://semver.org/s
 
 #### Blueprint
 - Added `conduit::blueprint::mesh::matset::to_silo()` which converts a valid blueprint matset to a node that contains arrays that follow Silo's sparse mix slot volume fraction representation.
+- Added `conduit::blueprint::mesh::field::to_silo()` which converts a valid blueprint field and matset to a node that contains arrays that follow Silo's sparse mix slot volume fraction representation.
+- Added `material_map` to `conduit::blueprint::mesh:matset::index`, to provide an explicit material name to id mapping.
+- Added `mat_check` field to `blueprint::mesh::examples::venn`. This field encodes the material info in a scalar field and in the `matset_values` in a way that can be used to easily compare and verify proper construction in other tools.
+
+### Fixed
+
+#### Relay
+- Fixed bug in the Relay IOHandle Basic that would create unnecessary "_json" schema files to be written to disk upon open().
 
 
 ## [0.6.0] - Released 2020-11-02

src/docs/sphinx/blueprint_mesh.rst

@@ -425,12 +425,12 @@ Uni-Buffer Material Sets
 *********************************
 
 A **uni-buffer** material set is one that presents all of its volume fraction data in a single data buffer.
-In this case, the material set schema must include this volume fraction data buffer, a parallel buffer associating each volume with a material identifier, and an *Object* mapping of human-readable material names to each unique material identifier.
+In this case, the material set schema must include this volume fraction data buffer, a parallel buffer associating each volume with a material identifier, and an *Object* that maps human-readable material names to unique integer material identifiers.
 Additionally, the top-level of this schema is an **o2mrelation** that sources from the volume fraction/material identifier buffers and targets the material topology.
 To conform to protocol, each ``matsets`` child of this type must be an *Object* that contains the following information:
 
    * matsets/matset/topology: "topo"
-   * matsets/matset/material_map: (integer object)
+   * matsets/matset/material_map: (object with integer leaves)
    * matsets/matset/material_ids: (integer array)
    * matsets/matset/volume_fractions: (floating-point array)
 
@@ -465,11 +465,15 @@ Multi-Buffer Material Sets
 
 A **multi-buffer** material set is a material set variant wherein the volume fraction data is split such that one buffer exists per material.
 The schema for this variant dictates that each material be presented as an *Object* entry of the ``volume_fractions`` field with the material name as the entry key and the material volume fractions as the entry value.
+**Multi-buffer** material sets also support an optional ``material_map``, which is an *Object* that maps human-readable material names to unique integer material identifiers. 
+If omitted, the map from material names to ids is inferred from the order of the material names in the ``volume_fractions`` node.
+
 Optionally, the value for each such entry can be specified as an **o2mrelation** instead of a flat array to enable greater specification flexibility.
 To conform to protocol, each ``matsets`` child of this type must be an *Object* that contains the following information:
 
    * matsets/matset/topology: "topo"
    * matsets/matset/volume_fractions: (object)
+   * matsets/matset/material_map: (optional, object with integer leaves)
 
 The following diagram illustrates a simple **multi-buffer** material set example:
 
@@ -493,6 +497,9 @@ The following diagram illustrates a simple **multi-buffer** material set example
             b:
               values: [0, b0, b2, b1, 0]
               indices: [1, 3, 2]
+          material_map: # (optional)
+            a: 0
+            b: 1
 
 
 Material Set Indexing Variants
@@ -529,6 +536,10 @@ The following diagram illustrates a simple **element-dominant** material set exa
             a: [a0, a1, 0]
             b: [b0, b1, b2]
             c: [0, 0, c2]
+          material_map: # (optional)
+            a: 0
+            b: 1
+            c: 2
 
 
 Material-Dominant Material Sets
@@ -562,6 +573,10 @@ The following diagram illustrates a simple **material-dominant** material set ex
             a: [0, 1]
             b: [0, 1, 2]
             c: [2]
+          material_map: # (optional)
+            a: 0
+            b: 1
+            c: 2
 
 
 Fields

src/libs/blueprint/conduit_blueprint_mesh.cpp

@@ -2670,17 +2670,16 @@ mesh::generate_index(const Node &mesh,
             idx_matset["topology"] = matset["topology"].as_string();
 
             // support different flavors of valid matset protos
+            //
+            // if we have material_map (node with names to ids)
+            // use it in the index
             if(matset.has_child("material_map"))
             {
-                NodeConstIterator mats_itr = matset["material_map"].children();
-                while(mats_itr.has_next())
-                {
-                    mats_itr.next();
-                    idx_matset["materials"][mats_itr.name()];
-                }
+                idx_matset["material_map"] = matset["material_map"];
             }
             else if(matset.has_child("materials"))
             {
+                // NOTE: I believe path is deprecated ... 
                 NodeConstIterator mats_itr = matset["materials"].children();
                 while(mats_itr.has_next())
                 {
@@ -2690,11 +2689,14 @@ mesh::generate_index(const Node &mesh,
             }
             else if(matset.has_child("volume_fractions"))
             {
+                // we don't have material_map (node with names to ids)
+                // so mapping is implied from node order, construct
+                // an actual map that follows the implicit order
                 NodeConstIterator mats_itr = matset["volume_fractions"].children();
                 while(mats_itr.has_next())
                 {
                     mats_itr.next();
-                    idx_matset["materials"][mats_itr.name()];
+                    idx_matset["material_map"][mats_itr.name()] = mats_itr.index();
                 }
             }
             else // surprise!
@@ -4511,13 +4513,49 @@ mesh::topology::shape::verify(const Node &shape,
 // blueprint::mesh::matset protocol interface
 //-----------------------------------------------------------------------------
 
+//-----------------------------------------------------------------------------
+// helper to verify a matset material_map
+//-----------------------------------------------------------------------------
+bool verify_matset_material_map(const std::string &protocol,
+                                const conduit::Node &matset,
+                                conduit::Node &info)
+{
+    bool res = verify_object_field(protocol, matset, info, "material_map");
+
+    if(res)
+    {
+        // we already know we have an object, children should be 
+        // integer scalars
+        NodeConstIterator itr = matset["material_map"].children();
+        while(itr.has_next())
+        {
+            const Node &curr_child = itr.next();
+            if(!curr_child.dtype().is_integer())
+            {
+                log::error(info,
+                           protocol,
+                           log::quote("material_map") +
+                           "child " +
+                           log::quote(itr.name()) +
+                           " is not an integer leaf.");
+                res = false;
+            }
+        }
+    }
+
+    log::validation(info, res);
+
+    return res;
+}
+
 //-----------------------------------------------------------------------------
 bool
 mesh::matset::verify(const Node &matset,
                      Node &info)
 {
     const std::string protocol = "mesh::matset";
     bool res = true, vfs_res = true;
+    bool mat_map_is_optional = true;
     info.reset();
 
     res &= verify_string_field(protocol, matset, info, "topology");
@@ -4535,12 +4573,11 @@ mesh::matset::verify(const Node &matset,
             verify_number_field(protocol, matset, info, "volume_fractions"))
         {
             log::info(info, protocol, "detected uni-buffer matset");
+            // materials_map is not optional in this case, signal
+            // for opt check down the line
+            mat_map_is_optional = false;
 
             vfs_res &= verify_integer_field(protocol, matset, info, "material_ids");
-            // TODO(JRC): Add a more in-depth verifier for 'material_map' that
-            // verifies that it's one level deep and that each child child houses
-            // an integer-style array.
-            vfs_res &= verify_object_field(protocol, matset, info, "material_map");
             vfs_res &= blueprint::o2mrelation::verify(matset, info);
 
             res &= vfs_res;
@@ -4574,6 +4611,47 @@ mesh::matset::verify(const Node &matset,
         }
     }
 
+    if(!mat_map_is_optional && !matset.has_child("material_map"))
+    {
+        log::error(info, protocol,
+            "'material_map' is missing (required for uni-buffer matsets) ");
+        res &= false;
+    }
+
+    if(matset.has_child("material_map"))
+    {
+        if(mat_map_is_optional)
+        {
+            log::optional(info, protocol, "includes material_map");
+        }
+
+        res &= verify_matset_material_map(protocol,matset,info);
+
+        // for cases where vfs are an object, we expect the material_map child 
+        // names to be a subset of the volume_fractions child names
+        if(matset.has_child("volume_fractions") &&
+           matset["volume_fractions"].dtype().is_object())
+        {
+            NodeConstIterator itr =  matset["material_map"].children();
+            while(itr.has_next())
+            {
+                itr.next();
+                std::string curr_name = itr.name();
+                if(!matset["volume_fractions"].has_child(curr_name))
+                {
+                    std::ostringstream oss;
+                    oss << "'material_map' hierarchy must be a subset of "
+                           "'volume_fractions'. " 
+                           " 'volume_fractions' is missing child '"
+                           << curr_name 
+                           <<"' which exists in 'material_map`" ;
+                    log::error(info, protocol,oss.str());
+                    res &= false;
+                }
+            }
+        }
+    }
+
     if(matset.has_child("element_ids"))
     {
         bool eids_res = true;
@@ -4675,7 +4753,18 @@ mesh::matset::index::verify(const Node &matset_idx,
     // performed on the "materials" field.
 
     res &= verify_string_field(protocol, matset_idx, info, "topology");
-    res &= verify_object_field(protocol, matset_idx, info, "materials");
+
+    // 2021-1-29 cyrush:
+    // prefer new "material_map" index spec, vs old "materials"
+    if(matset_idx.has_child("material_map"))
+    {
+        res &= verify_matset_material_map(protocol,matset_idx,info);
+    }
+    else
+    {
+        res &= verify_object_field(protocol, matset_idx, info, "materials");
+    }
+
     res &= verify_string_field(protocol, matset_idx, info, "path");
 
     log::validation(info, res);

src/libs/blueprint/conduit_blueprint_mesh.hpp

@@ -414,7 +414,7 @@ namespace matset
     // For details about the silo format, see documentation for 
     // 'DBPutMaterial' at:
     // https://wci.llnl.gov/content/assets/docs/simulation/computer-codes/silo/LLNL-SM-654357.pdf
-    void CONDUIT_BLUEPRINT_API to_silo(const conduit::Node &n,
+    void CONDUIT_BLUEPRINT_API to_silo(const conduit::Node &matset,
                                        conduit::Node &dest,
                                        const float64 epsilon = CONDUIT_EPSILON);
 
@@ -441,6 +441,19 @@ namespace field
     bool CONDUIT_BLUEPRINT_API verify(const conduit::Node &n,
                                       conduit::Node &info);
 
+    //-------------------------------------------------------------------------
+    // Given a blueprint field and matset, converts the matset and the field
+    // values + matset_values to the silo style sparse mixed slot
+    // representation.
+    //
+    // For details about the silo format, see documentation for 
+    // 'DBPutZZZVar' methods `mixvar` / `mixlen` params at:
+    // https://wci.llnl.gov/content/assets/docs/simulation/computer-codes/silo/LLNL-SM-654357.pdf
+    void CONDUIT_BLUEPRINT_API to_silo(const conduit::Node &field,
+                                       const conduit::Node &matset,
+                                       conduit::Node &dest,
+                                       const float64 epsilon = CONDUIT_EPSILON);
+
     //-------------------------------------------------------------------------
     // blueprint::mesh::field::index protocol interface
     //-------------------------------------------------------------------------