Add JGF reader support #521

dongahn · 2019-09-29T22:39:52Z

This PR adds JGF reader support. This should not be merged before PR #520.

Introduce reader classes into the new resource/readers directory and refactor resource_graph_db_t into its own directory (resource/store). Reader classes not only refactor the existing reader code (i.e., GRUG and Hwloc) but also add the new JGF reader code.
Integrate the new reader support into resource-query and resource module.
As part of that, change the command line options for resource-query and the load options for resource module. For resource-query, --grug and --hwloc-xml are deprecated and --load-file and --load-format are added instead. --hwloc-whitelist is now changed to --load-whitelist. For the resource module, grug-conf, hwloc-xml, and hwloc-whitelist have been deprecated. Instead, load-file, load-format, and load-whitelist are used.
Adjust the existing tests for these option name changes.
Add three new test cases to ensure the correctness of the JGF reader.

dongahn · 2019-09-30T21:12:35Z

t3014-resource-basic-jgf.t tests are failing in the Travis CI. Let me look.

codecov-io · 2019-09-30T23:32:38Z

Codecov Report

Merging #521 into master will decrease coverage by 0.34%.
The diff coverage is 79%.

@@            Coverage Diff             @@
##           master     #521      +/-   ##
==========================================
- Coverage   76.38%   76.03%   -0.35%     
==========================================
  Files          59       69      +10     
  Lines        6221     6481     +260     
==========================================
+ Hits         4752     4928     +176     
- Misses       1469     1553      +84

Impacted Files	Coverage Δ
resource/utilities/command.hpp	`100% <ø> (ø)`	⬆️
resource/schema/resource_graph.hpp	`59.25% <ø> (-2.81%)`	⬇️
resource/readers/resource_spec_grug.hpp	`100% <ø> (ø)`
resource/store/resource_graph_store.cpp	`100% <100%> (ø)`
resource/readers/resource_reader_grug.hpp	`100% <100%> (ø)`
resource/readers/resource_reader_jgf.hpp	`100% <100%> (ø)`
resource/utilities/command.cpp	`68.75% <100%> (ø)`	⬆️
resource/readers/resource_reader_base.hpp	`100% <100%> (ø)`
resource/store/resource_graph_store.hpp	`100% <100%> (ø)`
resource/readers/resource_reader_hwloc.hpp	`100% <100%> (ø)`
... and 20 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update adea512...b86a85f. Read the comment docs.

dongahn · 2019-10-01T02:10:59Z

Travis is green now. @SteVwonder, @tpatki and @milroy: this PR should also be ready for your reviews. This should just be merged after PR #520, though.

dongahn · 2019-10-01T16:46:08Z

PR #523 has been merged. This needs to be rebased to the upstream master.

dongahn · 2019-10-01T20:28:43Z

Rebased to the current master.

dongahn · 2019-10-02T22:31:48Z

Just rebased to the master with PR #520 and forced a push.

tpatki · 2019-10-04T02:00:19Z

resource/modules/resource_match.cpp

+    string load_file;              /* load file name */
+    string load_format;            /* load reader format */
+    string load_whitelist;         /* load resource whitelist */


@dongahn:
Shall provide a detailed review soon, but had a quick comment.

Can we change the variable names to reflect that this is for a graph? Maybe something like graph_input_file, graph_format is better than load_file and load_format?

Also, the whitelist is currently specific to hwloc. Do we expect any other future graph formats (beyond the three we have) to use the whitelist?

If not, I'm wondering if set_whitelist should be specific to the hwloc reader class as opposed to the base class. The current design is fine, I'm suggesting this to make the code easily readable, less verbose for the two readers that don't use it, and having a clearer check specific to hwloc reader here instead:

flux-sched/resource/modules/resource_match.cpp

Line 447 in f1d8b07

if (rd->set_whitelist (ctx->args.load_whitelist) < 0)

The variable names are matched with the command line options. There other graph related option in resource query so I didn't use graph. If you feel strongly about these, I will be happy to change them, though. Let me know.

Yes there is a good chance that white list can be supported by other readers and I made this a method in the base class for future extension.

tpatki · 2019-10-04T02:09:10Z

resource/modules/resource_match.cpp

+    if ( (rd = create_resource_reader (ctx->args.load_format)) == nullptr) {
+        flux_log (ctx->h, LOG_ERR, "creating load reader");


Space between parentheses.

Shouldn't the error message be "Can't create load reader"?

Also, a clear set of parentheses around the create_resource_reader (ctx->args.load_format)) == nullptr before assigning to rd will make the if statement more readable. Readability of if (return_val = function () == nullptr) format is confusing.

If you are talking about the extra whitespace in if ( (rd =, this is a pretty common practive. In fact the famous Steven book on "Unix system programming" recommends this. If this was defined in our style guide, I can change it. But I don't recall if I have seen this rewuirement.

Shouldn't the error message be "Can't create load reader"?

This was intentional as LOG_ERR generates "err" in the message prefix. But I can see why your message makes more sense. So I will change it. Thanks.

Also, a clear set of parentheses around the create_resource_reader (ctx->args.load_format)) == nullptr before assigning to rd will make the if statement more readable. Readability of if (return_val = function () == nullptr) format is confusing.

Hmmm yes that's why I had the extra parenthesis before rc and the end of the method call. Maybe I am missing something.

tpatki · 2019-10-04T02:39:28Z

resource/modules/resource_match.cpp

+        }
+        o = get_string_blocking (h, k);
+        hwloc_xml = json_string_value (o);
+        if ( (rc = ctx->db.load (hwloc_xml, rd, v, rank)) < 0) {


Space between parentheses.

tpatki · 2019-10-04T02:41:05Z

resource/modules/resource_match.cpp

+    }
+    buffer << in_file.rdbuf ();
+    in_file.close ();
+    if ( (rc = ctx->db.load (buffer.str (), rd)) < 0) {


Not sure if this is part of the style guide, but there's an extra space between the parenthesis here.

Please see above.

tpatki · 2019-10-04T02:47:44Z

resource/modules/resource_match.cpp

+    }
+    o = get_string_blocking (h, k);
+    hwloc_xml = json_string_value (o);
+    if ( (rc = ctx->db.load (hwloc_xml, rd, rank)) < 0) {


Space between parentheses.

tpatki · 2019-10-04T02:49:45Z

resource/modules/resource_match.cpp

+    if (ctx->args.load_whitelist != "") {
+        if (rd->set_whitelist (ctx->args.load_whitelist) < 0)
+            flux_log (ctx->h, LOG_ERR, "setting whitelist");
+        if (!rd->is_whitelist_supported ())
+            flux_log (ctx->h, LOG_WARNING, "whitelist unsupported");


Not sure if whitelisting should be a part of the base class (see earlier comment on whether we expect future graph formats other than hwloc to have whitelists).

Yes exactly. This is there in the base class for future extension.

tpatki · 2019-10-04T02:50:29Z

resource/modules/resource_match.cpp

-            rc = -1;
-            flux_log (ctx->h, LOG_ERR, "error in generating resources");
+        if (populate_resource_db_kvs (ctx, rd) < 0) {
+            flux_log (ctx->h, LOG_ERR, "loading resources from the KVS");


Maybe change the message to "Error loading resources from KVS"?

My understanding was "err" will be emitted with LOG_ERR and this was part of my effort to avoid redundancy. Let me double check though.

tpatki · 2019-10-04T03:17:16Z

resource/readers/resource_reader_grug.cpp

+    int get_rank ();
+
+private:
+    vtx_t emit_vertex (ggv_t u, gge_t e, const gg_t &recipe,


Change ggv_t to vtx_t?

Actually here v and e are an vertex and edge of the resource generation graph. So ggv_t should be correct I think.

tpatki · 2019-10-04T03:17:53Z

resource/readers/resource_reader_grug.cpp

+    ~dfs_emitter_t ();
+
+    void tree_edge (gge_t e, const gg_t &recipe);
+    void finish_vertex (ggv_t u, const gg_t &recipe);


Change ggv_t to vtx_t?

Same as above.

tpatki · 2019-10-04T03:18:19Z

resource/readers/resource_reader_grug.cpp

+    int gen_id (gge_t e, const gg_t &recipe,
+                int i, int sz, int j);
+
+    map<ggv_t, vector<vtx_t>> m_gen_src_vtx;


Change ggv_t to vtx_t?

tpatki · 2019-10-04T03:19:11Z

resource/readers/resource_reader_grug.cpp

+    g[e].name[recipe[ge].e_subsystem] = recipe[ge].rrelation;
+}
+
+vtx_t dfs_emitter_t::emit_vertex (ggv_t u, gge_t e, const gg_t &recipe,


ggv_t to vtx_t

tpatki · 2019-10-04T03:22:33Z

resource/readers/resource_reader_grug.cpp

+    int id = 0;
+
+    if (src_v == boost::graph_traits<resource_graph_t>::null_vertex ()) {
+        // ROOT!!


Minor: maybe change the comment to be // Root vertex of graph. as opposed to ROOT!!

Sure. I will change it.

tpatki · 2019-10-04T03:24:16Z

resource/readers/resource_reader_grug.cpp

+    vtx_t src_vtx, tgt_vtx;
+    ggv_t src_ggv = source (e, recipe);
+    ggv_t tgt_ggv = target (e, recipe);


I'm unsure about legitimate use of ggv_t versus vtx_t. Will revisit these once I understand this more clearly.

I left a comment to an issue ticket where you asked this question. Thanks

tpatki · 2019-10-04T03:28:25Z

resource/utilities/resource-query.cpp

-        ctx->params.grug = "conf/default";
-
-    gettimeofday (&st, NULL);
+    if ( (rd = create_resource_reader (ctx->params.load_format)) == nullptr) {


Space between parenthesis

Same as above.

tpatki · 2019-10-04T03:28:41Z

resource/utilities/resource-query.cpp


+    gettimeofday (&st, NULL);
+    if ( (rc = ctx->db.load (buffer.str (), rd)) != 0) {


Space between parenthesis.

tpatki · 2019-10-04T03:34:18Z

@dongahn: I made a few very minor comments, I will have to look at this in detail (esp the new JGF reader and tests) and I shall get to it soon.

tpatki · 2019-10-04T03:35:30Z

Also, this will need to be rebased. t3014 and t4007 were part of the variation-aware scheduler that was just merged, and those test cases will have to be renamed to a different number.

dongahn · 2019-10-04T04:06:31Z

Thanks @tpatki. I left my responses above. I will change a few things you spotted and as part of this I will adjust the test numbering that conflicts with your variation PR.

Refactor resource_graph_db_t into its own directory (resource/store). This will replace the same named class in resource/schema/resource_graph.hpp when integrated into the rest of the system. Add two load() methods into this class which takes in an object of the base reader class to decode the input string into its resource graph. How it is decoded is delegated down to the reader object whose runtime type must be a derived class of the base reader class. Introduce resource_graph_metadata_t that contains metadata about the resource graph. This helps simplify reader class methods. Add readers/resource_reader_base.[hpp|cpp] which define the base reader interfaces. They include a few pure virtual abstract interfaces that upcoming derived reader classes must override. The main methods of this class are unpack() and unpack_at(). The unpack() method deserializes the given string into a resource graph. The unpack_at() method unpacks the given string into a resource graph and then grafts the top-level vertices to an existing vertex in the resource graph.

dongahn · 2019-10-04T18:47:27Z

I just forced a push. I rebased this to the latest master with @tpatki's variation aware scheduling work. And I believe I have also addressed all the issues identified from the review discussions thus far.

@SteVwonder and/or @milroy : do you have other reviews?

milroy · 2019-10-04T19:36:47Z

@dongahn I am reviewing this PR now and will be finished with it today.

milroy

I added a few questions and nits regarding parentheses, but this PR looks good to me.

milroy · 2019-10-04T20:08:06Z

resource/store/resource_graph_store.hpp

+    std::map<subsystem_t, vtx_t> roots;
+    std::map<std::string, std::vector <vtx_t>> by_type;
+    std::map<std::string, std::vector <vtx_t>> by_name;
+    std::map<std::string, std::vector <vtx_t>> by_path;


by_path eventually should be changed to std::map<std::string, vtx_t> or the like (see issue #518). That's beyond the scope of this PR, but I'm noting it here for future reference. I'll try to address it in my upcoming PR based on issue #511.

Agreed! The changes like this are now lumped into either hardening and/or optimization project in my kanban board.

milroy · 2019-10-04T20:14:30Z

resource/readers/resource_reader_grug.cpp

@@ -0,0 +1,465 @@
+/*****************************************************************************\
+ *  Copyright (c) 2014 Lawrence Livermore National Security, LLC.  Produced at


I think this a new file, so the copyright should be 2019. That said, much of the content is from an older file, so perhaps the 2014 date is correct here.

I don't know what is the conversion when changes are made across multiple years. Like 2014 - 2019. I will change it that way and see how it looks.

milroy · 2019-10-04T20:37:24Z

resource/readers/resource_reader_grug.cpp

+    for ( ; ei != ee; ++ei) {
+        if (target (*ei, g) == tgt_v) {
+            e = (*ei);
+            return e;
+        }


Does this loop exist to avoid calling boost::edge (src_v, tgt_v, g) and its O(E/V) complexity for vecS edges?

Yes, once you find the matching edge, you don't need to further traverse out edges so you return.

As we discussed before, being able to find an edge with O(map) like performance will greatly be helpful.

Yes, that makes sense. I've higher performance lookups in my WIP on issue #511.

Great! You may want to split your WIP into multiple PRs and get some early reviews before it gets too large. It seems a faster edge lookup as a standalone topic can make a good enough PR. Also there are some performance benchmarks you can try your lookup logic to see how much it improves the performance.

milroy · 2019-10-04T20:39:30Z

resource/readers/resource_reader_grug.cpp

+            for (tgt_it = m.by_type[recipe[tgt_ggv].type].begin();
+                 tgt_it != m.by_type[recipe[tgt_ggv].type].end(); tgt_it++) {


Nit: .begin() and .end() need spaces after the parentheses.

Good catch! I will fix them.

milroy · 2019-10-04T20:39:38Z

resource/readers/resource_reader_grug.cpp

+            for (tgt_it = m.by_type[recipe[tgt_ggv].type].begin();
+                 tgt_it != m.by_type[recipe[tgt_ggv].type].end(); tgt_it++) {


Nit: .begin() and .end() need spaces after the parentheses.

milroy · 2019-10-04T20:46:50Z

resource/readers/resource_spec_grug.cpp

+            return (gen_meth_t)str2genmeth[i].e;
+    return (gen_meth_t)str2genmeth[i].e;


Should these casts have spaces after the close parentheses?

I don't think we have that required by our coding guide. My preference is not to have a space for casting.

milroy · 2019-10-04T20:48:00Z

resource/readers/resource_spec_grug.cpp

+        vtx_basename_map_t v_bn_map = get(&resource_pool_gen_t::basename, g);
+        vtx_size_map_t v_sz_map = get(&resource_pool_gen_t::size, g);
+        //vtx_unit_map_t v_ut_map = get(&resource_pool_gen_t::unit, g);
+        vtx_subsystem_map_t v_ss_map = get(&resource_pool_gen_t::subsystem, g);
+        edg_relation_map_t e_rel_map = get(&relation_gen_t::relation, g);
+        edg_gen_method_map_t e_gen_map = get(&relation_gen_t::gen_method, g);
+        edg_multi_scale_map_t e_ms_map = get(&relation_gen_t::multi_scale, g);


Change to get (...?

Good catch. I will fix those.

milroy · 2019-10-04T22:07:55Z

resource/readers/resource_reader_hwloc.cpp

+            if (strncmp(obj->name, "cuda", 4) == 0)
+                id = atoi (obj->name + 4);
+            else if (strncmp(obj->name, "opencl", 6) == 0) {


Nits: spaces after strncmp.

milroy · 2019-10-04T22:21:18Z

resource/readers/resource_reader_jgf.cpp

+    for (auto kv : g[v].paths) {
+        g[v].idata.member_of[kv.first] = "*";
+        m.by_path[kv.second].push_back (v);
+        if (std::count(kv.second.begin(), kv.second.end(), '/') == 1)


Spaces after begin and end.

Thanks. (Maybe we need some format enforcer :-).

dongahn · 2019-10-04T22:58:57Z

Thanks @milroy! I will address the issues above and push my changes.

milroy · 2019-10-07T06:25:33Z

resource/readers/resource_reader_base.hpp

+    /*! Unpack str into a resource graph and graft
+     *  the top-level vertices to vtx.
+     *
+     * \param g      resource graph
+     * \param m      resource graph meta data
+     * \param vtx    parent vtx at which to graft the deserialized graph
+     * \param str    resource set string
+     * \param rank   assign rank to all of the newly created resource vertices
+     * \return       0 on success; non-zero integer on an error
+     */
+    virtual int unpack_at (resource_graph_t &g, resource_graph_metadata_t &m,
+                           vtx_t &vtx, const std::string &str, int rank = -1) = 0;


param vtx will have edges to and/or from each of the subsystem roots of the unpacked resource graph, correct? Also, is vtx in the original graph but not in the unpacked graph?

@milroy: great question. Easy one first:

Also, is vtx in the original graph but not in the unpacked graph?

Yes, it is a vertex that should only be in the original graph. This interface is currently only wired with the hwloc reader to attach a new set of subgraph created by each of the ranked hwloc xml objects into the root vertex: the cluster vertex.

param vtx will have edges to and/or from each of the subsystem roots of the unpacked resource graph, correct?

The intent is broader than this.

Say, the unpacked graph is node10->core0. While node10 is the root of the subgraph, it is not the root of the overall graph. I detect the root of the subgraph by parsing each vertex's paths field. If node10 vertex's paths has containment: /cluster/node10, it won't be identified as the root of the containment subsystem.

Also if the unpacked graph is node10->core0, node20->core30, I expect that this interface attach both node10 and node20 to the vtx. (We don't currently have this use case as hwloc reader reads one rank at a time.) If this were to work, it is almost like the caller should specify what those vertices are, which need to be attached to vtx. ..

If you want to propose something more general than this interface, which can help your work, maybe we can work on an interface revision after this PR lands?

This interface is currently only wired with the hwloc reader to attach a new set of subgraph created by each of the ranked hwloc xml objects into the root vertex: the cluster vertex.

But the intention is that eventually vtx can be any vertex in the original graph?

While node10 is the root of the subgraph, it is not the root of the overall graph. I detect the root of the subgraph by parsing each vertex's paths field.

That's sensible. I was abusing the term "subsystem roots" as referring to the top-level node(s) in the unpacked graph (the subgraph). Can the subgraph have multiple roots (i.e. the subgraph contains vertices in multiple subsystems)?

Also if the unpacked graph is node10->core0, node20->core30, I expect that this interface attach both node10 and node20 to the vtx. (We don't currently have this use case as hwloc reader reads one rank at a time.) If this were to work, it is almost like the caller should specify what those vertices are, which need to be attached to vtx. ..

This is my main curiosity. The path to vertex mapping is many-to-one since each vertex can have a path per subsystem: g[v].paths[subsys] = prefix + "/" + g[v].name;. This suggests to me that node10 could be attached to one vtx from each subsys where g[node10].paths[subsys] is defined.

If you want to propose something more general than this interface, which can help your work, maybe we can work on an interface revision after this PR lands?

Yes, I think we will need a more general interface which accepts a node to vertex map. I agree, that work is out of scope for this PR.

Yes, I think we will need a more general interface which accepts a node to vertex map. I agree, that work is out of scope for this PR.

Do you want to track the idea with a new issue?

milroy · 2019-10-08T22:50:20Z

@dongahn it looks like you still need to add spaces after a few function calls. After that I can merge this PR, per our discussion with @SteVwonder.

dongahn · 2019-10-08T23:08:07Z

@dongahn it looks like you still need to add spaces after a few function calls. After that I can merge this PR, per our discussion with @SteVwonder.

Yes, I planned to do this after some urgent tasks. Stay tuned.

dongahn · 2019-10-09T05:05:44Z

@milroy: OK, I pushed all the requested changes. Once you give a quickly review on those intermediate commits, I will squash them before this gets merged. Let me know.

milroy · 2019-10-09T06:18:47Z

@dongahn looks good! The copyright year range is a good solution.

Introduce resource_reader_grug_t by deriving it from resource_reader_base_t. Refactor GRUG reader code into resource_reader_grug_t and implement the unpack() method. Adjust this code to wean off of the old resource_graph_db_t and make use of resource_graph_t and resource_graph_metadata_t instead. The unpack_at() method is unsupported. Resource whitelist is not supported either, and this is indicated by returning false from is_whitelist_supported() method.

Introduce resource_reader_hwloc_t by deriving it from resource_reader_base_t. Refactor Hwloc reader code into resource_reader_hwloc_t and implement the unpack() and unpack_at() methods. Adjust this code to wean off of the old resource_graph_db_t and make use of resource_graph_t and resource_graph_metadata_t instead. Add a bit better error message support for hwloc API calls from the original hwloc reader code.

Introduce resource_reader_jgf_t by deriving it from resource_reader_base_t. First, fetch each JGF node from the "nodes" key from the JGF string, and add a vertex of the resource graph data while building vmap that keeps track of the JGF nodeid-vertex pair using that info. Next, fetch each JGF node from the "edges" key and add an edge of the resource graph data using that info in combination with the source and target vertices provided by the vmap. As part of resource graph vertex and edge population, we fill in graph infrastructure information such as idata.member_of as well as resource graph metadata (e.g., roots of each detected subsystem). Detect each unique subsystem and its root vertex by using the paths field of each JGF node. If the paths field of a JGF node contains a path with one slash (e.g., "containment": "/cluster0"), deem it as the root of that subsystem. Implement unpack() method while leaving unpack_at() method unsupported. Hence, is_whitelist_supported() returns false.

Instantiate an object of a derived resource reader class that corresponds to the input string (reader type name) and return it as the object of the resource reader base class. Allow the client code to be written as the base class, making it future proof. Use shared_ptr class to enhance safety.

Integrate the new resource reader scheme into the entire resource infastructure and the existing tests. - Drop the old reader/generator code in generators/gen.[hpp|cpp]. - Delete the resource_graph_db_t class from schema/resource_graph.hpp as the new definitions in store/resource_graph_store.[hpp|cpp] will be wired in. - Update resource/Makefile.am with respect to the above removals and the new resource reader files that were previously added. Allow the new code to make into the resource convenient library. Add jansson as an extra dependency as required by the JGF reader as well. - Change the command line options for resource-query and the load options for resource match module. For resource-query, --grug and --hwloc-xml are deprecated and --load-file and --load-format are added instead. --hwloc-whitelist is now changed to --load-whitelist. For the resource module, grug-conf, hwloc-xml, and hwloc-whitelist have been deprecated. Instead, load-file, load-format, and load-whitelist are used. - Adjust resource-query and resource module code with respect to the new resource_graph_db_t and resource_graph_metadata_t definitions. - populate_resource_db in resource-query and resource module code now makes use of the new resource reader support. - Update all of the sharness test cases to use the new cli and module-load options.

Add 3 machine definitions written in JGF in data/resource/jgfs (in parallel to data/resource/grugs). Will be used by future JGF reader sharness tests. hwloc_4core.json contains a very small machine configuration with a total of 1 compute node with 4 compute cores. power.json describes the same system specified by data/resource/grugs/power.graphml. tiny.json describes the same system specified by data/resource/grugs/tiny.graphml When power.json or tiny.json is loaded into resource-query and resource module, these programs must behave exactly same as they were loaded with the corresponding GRUG files. In fact, future test codes will be written based ont this property.

Copied from t3001-resource-basic.t. Only change the machine specification from GRUG to JGF. Because we use the exactly same resource-query commands, the test outputs must match with the results of t3001-resource-basic.t.

Copied from t3010-resource-power.t. Only change the machine specification from GRUG to JGF. Because we use the exactly same resource-query commands, the test outputs must match with that of t3010-resource-power.t. Adjust this test code to ensure the JGF reader works correctness when there are multiple subsystems in the JGF spec (i.e. containment and power).

Add simple tests to ensure the JGF reader works when used within the resource module as well.

dongahn · 2019-10-09T17:10:05Z

@milroy: I squashed all the intermediate commits. Once Travis turns green, please go ahead and merge. Thank you @milroy, @tpatki and @SteVwonder for the review.

dongahn force-pushed the readers branch from eb8386a to aa7f238 Compare September 30, 2019 23:32

dongahn requested review from SteVwonder, milroy and tpatki September 30, 2019 23:50

dongahn mentioned this pull request Sep 30, 2019

Tidy up JGF match writer support #520

Merged

dongahn force-pushed the readers branch from aa7f238 to 59dee20 Compare October 1, 2019 20:28

This was referenced Oct 1, 2019

ResourceQuery returned value #510

Closed

resource: introduce class hierarchy for resource generator #411

Closed

dongahn force-pushed the readers branch 2 times, most recently from 1f1f403 to f1d8b07 Compare October 2, 2019 22:29

tpatki reviewed Oct 4, 2019

View reviewed changes

dongahn force-pushed the readers branch from f1d8b07 to 0fbc391 Compare October 4, 2019 18:44

milroy approved these changes Oct 4, 2019

View reviewed changes

milroy reviewed Oct 7, 2019

View reviewed changes

dongahn added 10 commits October 9, 2019 09:57

test: correctness of resource-query with the JGF reader

bf23c52

Copied from t3001-resource-basic.t. Only change the machine specification from GRUG to JGF. Because we use the exactly same resource-query commands, the test outputs must match with the results of t3001-resource-basic.t.

test: resource module works with the JGF reader

550b3ff

Add simple tests to ensure the JGF reader works when used within the resource module as well.

build: integrate 3 JGF tests into the build system

b86a85f

dongahn force-pushed the readers branch from e6bee55 to b86a85f Compare October 9, 2019 17:09

dongahn mentioned this pull request Oct 9, 2019

Generalize unpack_at() method within resource reader #533

Closed

milroy merged commit 16e72c3 into flux-framework:master Oct 9, 2019

		if ( (rd = create_resource_reader (ctx->args.load_format)) == nullptr) {
		flux_log (ctx->h, LOG_ERR, "creating load reader");


		gettimeofday (&st, NULL);
		if ( (rc = ctx->db.load (buffer.str (), rd)) != 0) {

		@@ -0,0 +1,465 @@
		/*****************************************************************************\
		* Copyright (c) 2014 Lawrence Livermore National Security, LLC. Produced at

		for (tgt_it = m.by_type[recipe[tgt_ggv].type].begin();
		tgt_it != m.by_type[recipe[tgt_ggv].type].end(); tgt_it++) {

		return (gen_meth_t)str2genmeth[i].e;
		return (gen_meth_t)str2genmeth[i].e;

Add JGF reader support #521

Add JGF reader support #521

Conversation

dongahn commented Sep 29, 2019

dongahn commented Sep 30, 2019

codecov-io commented Sep 30, 2019 • edited Loading

Codecov Report

dongahn commented Oct 1, 2019

dongahn commented Oct 1, 2019

dongahn commented Oct 1, 2019

dongahn commented Oct 2, 2019

tpatki Oct 4, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tpatki Oct 4, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tpatki Oct 4, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tpatki commented Oct 4, 2019

tpatki commented Oct 4, 2019

dongahn commented Oct 4, 2019

dongahn commented Oct 4, 2019

milroy commented Oct 4, 2019

milroy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dongahn commented Oct 4, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

milroy Oct 7, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

milroy commented Oct 8, 2019

codecov-io commented Sep 30, 2019 •

edited

Loading

tpatki Oct 4, 2019 •

edited

Loading

tpatki Oct 4, 2019 •

edited

Loading

tpatki Oct 4, 2019 •

edited

Loading

milroy Oct 7, 2019 •

edited

Loading