enh: add shell namespaces and verify for mesh bp #55

cyrush · 2016-09-20T23:23:05Z

added implementation of mesh/coordset verify

added implementation of mesh verify that relies on:
mesh/coordset (implemented)
mesh/topology (unimplemented)
mesh/field (unimplemented)

added info output to mcarray verify

wired generic interface for path based protocol identification

ex:
blueprint::verify(n,"mesh/coordset")
maps to
blueprint::mesh::coordset::verify(n,"mesh/coordset")

still evolving structure of the info output for verify, but
it looks like the following:

{
valid = "true" or "false"
(if errors)
errors = ["error string 1", "error string 2"]

child
{
valid = "true" or "false"
(if errors)
errors = ["error string 1", "error string 2"]
}
...
}

this allows us to report errors at any part in a hierarchy, but
assumes protocols won't require the names "valid" or "errors"

added implementation of mesh/coordset verify added implementation of mesh verify that relies on: mesh/coordset (implemented) mesh/topology (unimplemented) mesh/field (unimplemented) added info output to mcarray verify wired generic interface for path based protocol identification ex: blueprint::verify(n,"mesh/coordset") maps to blueprint::mesh::coordset::verify(n,"mesh/coordset") still evolving structure of the info output for verify, but it looks like the following: { valid = "true" or "false" (if errors) errors = ["error string 1", "error string 2"] child { valid = "true" or "false" (if errors) errors = ["error string 1", "error string 2"] } ... } this allows us to report errors at any part in a hierarchy, but assumes protocols won't require the names "valid" or "errors"

xjrc · 2016-09-21T17:12:22Z

src/libs/blueprint/blueprint_mcarray.cpp

+    bool res = true;
+
+    // mcarray needs to be an object or a list
+    if( ! (n.dtype().is_object() || n.dtype().is_list()) )


This is an incredibly minor complaint, but I feel like this line doesn't match the coding standards used for of the rest of Conduit. In particular, it doesn't feel like the expanded spacing in this statement fits; something like if(!(n.dtype().is_object() || n.dtype().is_list())) seems more appropriate.

sounds good

xjrc · 2016-09-21T17:49:31Z

src/libs/blueprint/blueprint_mesh.cpp

+    }
+
+    // required: "topologies",  with at least one child
+    // each child must conform to "mesh::topology"


The functionality below for verifying the topologies path is nearly identical to that used above to verify the coordsets path. The commonalities between these two segments should be factored out if possible.

thats a good observation, if we can condense these checks into reusable functions that sounds like a good improvment

xjrc · 2016-09-21T17:50:37Z

src/libs/blueprint/blueprint_mesh.cpp

+        }
+    }
+
+    // optional: "fields", each child must conform to "mesh::field"


The functionality used below to verify the fields path is also nearly identical to the coordsets path verification functionality and should be considered during factoring.

It may be worth noting in the info node that this section is optional as is done with the extra dimensions for nodes in the coordset::verify_coordset_uniform function below.

yes, we should add an info entry to optional indicating that fields exist.

xjrc

Given how much duplicated/specifically factored code we need in order to implement these verification functions, I think that it may be better to tackle this implementation by leveraging rapidjson instead. We can use rapidjson to perform a lot of the basic schema validation (see here) and then just implement some of our more advanced requirements on top of this. This approach has a number of benefits, the most important being that (1) it tremendously reduces the amount of code we need to write and (2) it makes it easier to see/modify the mesh schema structure (see some of Spack's schemas for examples).

Does this alternate method make sense and/or seem feasible? If it doesn't make sense, please let me know and I'll try to lay out what I'm thinking more clearly.

xjrc · 2016-09-21T18:09:33Z

src/libs/blueprint/blueprint_mesh.cpp

+            info["errors"].append().set(msg);
+            res = false;
+        }
+        else if(!dims["i"].dtype().is_number())


All of these dimensional value verification procedures have very similar code as well and should be factored if possible.

One of my goals is for the the code used to verify the protocol to reflect how the data will be consumed.

Given this, I don't think we should use rapidjson here b/c that would require transforming the data (or its schema) into json. At this point the data is not json -- it's in conduit's in memory format.

Even with an exemplar schema, or access to standard set of schema validation primitives I suspect we will still have to have some custom code for cases where schema validation doesn't get us all the way there. In that case we have two styles of validation which will be harder to maintain.

We could try something similar in conduit (compare to exemplar nodes), but like schema validation I don't think that will cover all of what is admissible.

You certainly bring up a lot of valid points, but I still think that leveraging rapidjson here will help us a lot more than it will hinder us. If you don't mind, could we discuss this further during our meeting tomorrow afternoon? If you're not convinced after I try to elaborate more on why I think this is a good idea, I won't pursue this any further; I just want to advocate a bit more for this idea because I think it could save us a lot of work.

sure, feel free to explore it more. I don't know if we will have too much time to get into the details tomorrow, but if you have a prototype of how it would look that would help frame the discussion.

xjrc · 2016-09-21T19:20:04Z

One quick addendum to my previous comment: using the schema validation functionality in rapidjson will require upgrading to version 1.1.0 (up from the current version of 1.0.2). If this isn't feasible, then just disregard my previous suggestion.

coveralls · 2016-09-21T23:34:13Z

Coverage increased (+0.01%) to 76.668% when pulling f32e1e1 on enh/2016_09_blueprint_mesh_verify into 16b0216 on master.

coveralls · 2016-09-30T22:52:52Z

Coverage increased (+0.2%) to 76.882% when pulling 981a4a8 on enh/2016_09_blueprint_mesh_verify into 16b0216 on master.

coveralls · 2016-10-01T00:13:12Z

Coverage increased (+0.2%) to 76.896% when pulling 808ddf8 on enh/2016_09_blueprint_mesh_verify into 16b0216 on master.

cyrush · 2016-10-01T06:16:40Z

index gen commit will resolve #38

also: refactored verify logging logic + helpers added check for grid_function to bp::mesh::generate_index

coveralls · 2016-10-03T21:33:54Z

Coverage increased (+0.2%) to 76.896% when pulling 3e6183f on enh/2016_09_blueprint_mesh_verify into 16b0216 on master.

cyrush · 2016-10-03T21:37:08Z

@xjrc -- this is coming along. The last piece is mesh:topology::verify. I may take a pass at it, but I will still need lots of help with testing.

Also wondering if how I am doing the nested namespaces in the .cpp file is confusing.
Since most of the functions are "verify", it may be better to drop the namespace statements and simply write something like:

bool
blueprint::mesh::topology::verify(const Node &n, Node &info)
{
}

instead of

namespace mesh
{
namespace toplogy
{
bool
verify(const Node &n, Node &info)
{
}
..
}
}

Thoughts?

xjrc · 2016-10-03T23:50:16Z

@cyrush: Even though the first version is more verbose, I like it better because I can easily just read the function title and know exactly which verify function I'm reading. It doesn't seem like eliminating the namespace scopes will be too detrimental anyway since each namespace is relatively sparsely populated.

I'll try to take a look at this sometime later this week. I'm pretty busy preparing for my presentation in a few weeks, but I'm going to try to be done by Friday. After I'm finished, I'll start contributing some test cases.

cyrush · 2016-10-03T23:52:33Z

thanks, I agree -- I'll make changes to the how namespaces are used in the cpp file.

remove none info variants of verify funcs

refactor bp verify sub cases into actual protocols

added more structure to coord_system protocol to support cart, cylnd, and spherical

coveralls · 2016-10-10T02:36:48Z

Changes Unknown when pulling d213787 on enh/2016_09_blueprint_mesh_verify into * on master*.

…m/llnl/conduit into enh/2016_09_blueprint_mesh_verify

coveralls · 2016-10-10T23:55:12Z

Changes Unknown when pulling 15b5685 on enh/2016_09_blueprint_mesh_verify into * on master*.

coveralls · 2016-10-11T03:56:21Z

Changes Unknown when pulling 6b0897b on enh/2016_09_blueprint_mesh_verify into * on master*.

cyrush · 2016-10-11T04:05:03Z

@xjrc this should include all of the additions for mesh bp verify (#40) and index gen ( #38 )

#49 will require a good amount of changes to the new verify code -- I am wondering if we should merge this PR, and start a new branch / PR focused on unit tests to improve coverage of the new code (and any fixes we find) (https://coveralls.io/builds/8274435/source?filename=src%2Flibs%2Fblueprint%2Fblueprint_mesh.cpp). That way I can easily move in parallel to resolve #49. What do you think?

xjrc · 2016-10-13T17:53:11Z

@cyrush: I agree; I think that merging this branch will make it much easier to move forward with testing. I'm going to start writing some tests today, but I'll hold off on pushing them to a public branch until this is merged to make future merges simpler.

cyrush · 2016-10-13T17:59:16Z

@xjrc merged this pr, feel free to create a new branch and open a new pr

xjrc · 2016-10-13T21:51:34Z

src/libs/blueprint/blueprint_mesh.cpp

+bool
+check_cyln_coord_sys_axis_name(std::string &name)
+{
+    return ( name == "r" || name == "z");


@cyrush: Shouldn't cylindrical coordinates be expressed using 3 coordinates (i.e. r, z, and theta)? If we're only planning to use cylindrical to describe cylindrical profiles then there isn't an issue, but I think that the third coordinate will be necessary for fully describing 3D geometries.

rz is for 2D cylindrical, the "r" component collapses 2 comps from 3d.

all the sim codes I know about only support 2D-cylindrical -- for the 3D case the r would be broken out into two components.

cyrush added this to the 0.2.0 milestone Sep 20, 2016

xjrc reviewed Sep 21, 2016

View reviewed changes

add bp mesh index verify imp

981a4a8

enh add func to gen mesh bp index from mesh

808ddf8

add bg::mesh::field verify

3e6183f

also: refactored verify logging logic + helpers added check for grid_function to bp::mesh::generate_index

cyrush mentioned this pull request Oct 3, 2016

generate blueprint index from blueprint tree #38

Closed

cyrush added 8 commits October 4, 2016 09:16

change mesh bp imp to use namespaces in func defs

e5d0172

remove none info variants of verify funcs

add topo verify

8ffda2e

refactor bp verify sub cases into actual protocols

flush out coord_system protocol

0946e1c

added more structure to coord_system protocol to support cart, cylnd, and spherical

try adding blueprint to coveralls coverage check

694849a

fix conflict with master

8e442fa

don't name unused args

1555181

Merge branch 'master' into enh/2016_09_blueprint_mesh_verify

047b974

simplify mcarray bp interface

0c2418c

cyrush mentioned this pull request Oct 5, 2016

0.2.0 warnings pass #57

Closed

fix index_t to int32 cov warnings in mesh bp examples

d213787

cyrush added 2 commits October 10, 2016 16:50

rework coord_system gen and verify

e17cc00

Merge branch 'enh/2016_09_blueprint_mesh_verify' of https://github.co…

15b5685

…m/llnl/conduit into enh/2016_09_blueprint_mesh_verify

add gf check to mesh bp topo

6b0897b

cyrush merged commit 82085ad into master Oct 13, 2016

cyrush deleted the enh/2016_09_blueprint_mesh_verify branch October 13, 2016 17:59

cyrush mentioned this pull request Oct 13, 2016

add initial blueprint verify for mesh, mesh::topology and mesh::field protocols #40

Closed

xjrc mentioned this pull request Oct 13, 2016

tests: add test cases for mcarray/mesh verify functions #72

Merged

xjrc reviewed Oct 13, 2016

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enh: add shell namespaces and verify for mesh bp #55

enh: add shell namespaces and verify for mesh bp #55

cyrush commented Sep 20, 2016

xjrc Sep 21, 2016

cyrush Sep 21, 2016

xjrc Sep 21, 2016

cyrush Sep 21, 2016

xjrc Sep 21, 2016 •

edited

xjrc Sep 21, 2016

cyrush Sep 21, 2016

xjrc left a comment

xjrc Sep 21, 2016

cyrush Sep 21, 2016

xjrc Sep 21, 2016

cyrush Sep 21, 2016

xjrc commented Sep 21, 2016

coveralls commented Sep 21, 2016

coveralls commented Sep 30, 2016

coveralls commented Oct 1, 2016

cyrush commented Oct 1, 2016

coveralls commented Oct 3, 2016

cyrush commented Oct 3, 2016

xjrc commented Oct 3, 2016

cyrush commented Oct 3, 2016

coveralls commented Oct 10, 2016

coveralls commented Oct 10, 2016

coveralls commented Oct 11, 2016

cyrush commented Oct 11, 2016

xjrc commented Oct 13, 2016

cyrush commented Oct 13, 2016

xjrc Oct 13, 2016

cyrush Oct 13, 2016

enh: add shell namespaces and verify for mesh bp #55

enh: add shell namespaces and verify for mesh bp #55

Conversation

cyrush commented Sep 20, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xjrc Sep 21, 2016 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xjrc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xjrc commented Sep 21, 2016

coveralls commented Sep 21, 2016

coveralls commented Sep 30, 2016

coveralls commented Oct 1, 2016

cyrush commented Oct 1, 2016

coveralls commented Oct 3, 2016

cyrush commented Oct 3, 2016

xjrc commented Oct 3, 2016

cyrush commented Oct 3, 2016

coveralls commented Oct 10, 2016

coveralls commented Oct 10, 2016

coveralls commented Oct 11, 2016

cyrush commented Oct 11, 2016

xjrc commented Oct 13, 2016

cyrush commented Oct 13, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xjrc Sep 21, 2016 •

edited