implement convenience workflow for using distributed mesh #10623

rwcarlsen · 2018-01-26T17:17:28Z

A few parts of this implementation feel a bit hacky - suggestions on improving it are certainly welcome.

Fixes #10458. Note that the mesh splitting won't work right until we
get another libmesh update in :-(

moosebuild · 2018-01-26T17:46:23Z

Job Documentation on 59b8479 wanted to post the following:

View the site here

This comment will be updated on new commits.

permcody · 2018-01-26T17:34:09Z

framework/src/base/MooseApp.C

+// splits s with whitespace and commas as delimiters and converts the split elements to integers
+// returned in a vector.
+std::vector<unsigned int>
+splitstr(const std::string & s)


https://github.com/idaholab/moose/blob/devel/framework/include/utils/MooseUtils.h#L378

How convenient... I'll switch.

permcody · 2018-01-26T17:38:35Z

framework/src/base/MooseApp.C

@@ -562,20 +600,45 @@ MooseApp::runInputFile()

  _action_warehouse.executeAllActions();

-  if (isParamValid("mesh_only"))
+  if (getParam<std::string>("split_mesh") != "")


I'd recommend that you break out this new logic into a subroutine.

Also, you probably don't want to run all Actions when doing the splits. Building the equation system might be really expensive for instance. A much better design would be to run the subset of Actions needed to build the mesh and then do the splits.

I'll piggy back on mesh-only logic.

permcody · 2018-01-26T17:44:20Z

framework/src/base/MooseApp.C

+      "",
+      "comma-separated list of numbers of chunks to split the mesh into");
+
+  params.addCommandLineParam<std::string>("split_file",


Maybe this parameter should be pluralized? split_files?

Maybe, but the user only passes a single file name as the argument to that flag - so it is a bit weird no matter how you do the pluralization.

I think I'm ok with split_file

In MOOSE we call these file_bases. Perhaps that would be a better name (more in line with what users see in the Outputs block.

Possibly, but there is actually a file that gets created that is named exactly what the user types in there. As currently implemented, it is left to the user to type in the ".cpr" extension. This was done to make is easy for tab completion to work if the user is trying to use an already split mesh. I think it would be confusing to call it [bla]-base if we expect the user to type in a file extension.

permcody · 2018-01-26T17:49:47Z

framework/include/mesh/MooseMesh.h

+   * Returns the name of the mesh file read to produce this mesh if any or an empty string
+   * otherwise.
+   */
+  virtual std::string getFileName() const { return ""; }


This seems like a reasonable new API to add, but I know we have several other mesh formats that read files spread all over (there maybe another in the framework, there are several in the modules, etc). You should probably take a look and make them all override this new interface.

I'll take a look. Not sure what to do about the meshes that read multiple files though.

This is done.

permcody · 2018-01-26T17:50:26Z

framework/include/utils/MooseUtils.h

+ * Removes any file extension from the fiven string s (i.e. any ".[extension]" suffix of s) and
+ * returns the result.
+ */
+std::string stripExtension(const std::string & s);


https://github.com/idaholab/moose/blob/devel/framework/include/utils/MooseUtils.h#L120

That doesn't strip extensions - it splits file name from prefix dirpaths.

So the method called stripExtensions doesn't actually strip extensions 😛 ?!

No - the function you linked (splitFileName) doesn't strip extensions.

friedmud

Looking good - just a few things.

friedmud · 2018-01-26T17:55:00Z

framework/src/base/MooseApp.C

+// splits s with whitespace and commas as delimiters and converts the split elements to integers
+// returned in a vector.
+std::vector<unsigned int>
+splitstr(const std::string & s)


We already have MooseUtils::tokenize() that does this.

friedmud · 2018-01-26T17:55:41Z

framework/src/base/MooseApp.C

@@ -562,20 +600,45 @@ MooseApp::runInputFile()

  _action_warehouse.executeAllActions();

-  if (isParamValid("mesh_only"))
+  if (getParam<std::string>("split_mesh") != "")


Why not isParamValid()?

That always returns true for parameters that have a default. I suppose split_mesh doesn't need a default though - I'll take a look.

friedmud · 2018-01-26T17:56:05Z

framework/src/base/MooseApp.C

+    auto & mesh = _action_warehouse.mesh();
+
+    if (mesh->getFileName() == "" && getParam<std::string>("split_file") == "")
+      mooseError("must specify an output mesh file name (--split-file) when splitting "


friedmud · 2018-01-26T17:57:02Z

framework/src/base/MooseApp.C

+      processor_id_type n = splits[i];
+      Moose::out << "splitting " << n << " ways..." << std::endl;
+
+      auto cpr = libMesh::split_mesh(*mesh, n);


Does the reset to 1 partition happen inside split_mesh()?

Also: maybe the libMesh interface should really just take the vector of splits... and do this loop internally.

Yes. I thought about that, but this allows the callers of the API more flexibility on how they want to handle the checkpoint object - they can switch it into ascii mode, etc. without having to pass a bajillion options into the splitting API. The goal was to make the libmesh splitting API do as close to one thing as possible and still preserve flexibility for callers.

friedmud · 2018-01-26T17:57:23Z

framework/src/base/MooseApp.C

+    for (std::size_t i = 0; i < splits.size(); i++)
+    {
+      processor_id_type n = splits[i];
+      Moose::out << "splitting " << n << " ways..." << std::endl;


Capitalize s?

permcody

Looking pretty good.

permcody · 2018-02-01T15:19:51Z

framework/src/actions/SetupMeshAction.C

+  if (_app.parameters().get<bool>("use_split"))
+  {
+    auto split_file = _app.parameters().get<std::string>("split_file");
+    if (_type != "FileMesh")


I still don't think this is right. We have several other FileMesh derived types. I know you haven't instantiated the object here yet so using a virtual method or some other normal trick doesn't work but we might be able to make this all more robust with the old validParams trick (embedding a "base" type in validParams).

permcody · 2018-02-01T15:20:44Z

framework/src/actions/SplitMeshAction.C

+  auto mesh = _app.actionWarehouse().mesh();
+
+  if (mesh->getFileName() == "" && _app.parameters().get<std::string>("split_file") == "")
+    mooseError("must specify an output mesh file name (--split-file) when splitting "


I know this isn't in the style guide but, we typically write sentences (start with a capital letter) in all of our error messaging throughout MOOSE - please change.

permcody · 2018-02-01T15:20:59Z

framework/src/actions/SplitMeshAction.C

+
+  auto splitstr = _app.parameters().get<std::string>("split_mesh");
+  std::vector<unsigned int> splits;
+  bool success = MooseUtils::tokenizeAndConvert(splitstr, splits, ", ");


permcody · 2018-02-01T15:22:25Z

framework/src/actions/SplitMeshAction.C

+    Moose::out << "splitting " << n << " ways..." << std::endl;
+
+    auto cpr = libMesh::split_mesh(*mesh, n);
+    Moose::out << "    * writing " << cpr->current_processor_ids().size() << " files per process..."


Note: OK, but we really should revisit the whole console stream interface so that we can use it more consistently. I know you can't here because your messages would be buffered.

permcody · 2018-02-01T15:23:14Z

framework/src/actions/SplitMeshAction.C

+  for (std::size_t i = 0; i < splits.size(); i++)
+  {
+    processor_id_type n = splits[i];
+    Moose::out << "splitting " << n << " ways..." << std::endl;


More lower case messages coming out here. We might need to have a discussion on this.

permcody · 2018-02-01T15:23:49Z

framework/src/base/MooseApp.C

+    {
+      _syntax.registerTaskName("split_mesh", true);
+      _syntax.addDependency("split_mesh", "setup_mesh_complete");
+      _action_warehouse.setFinalTask("split_mesh");


permcody · 2018-02-01T15:24:51Z

framework/src/base/MooseApp.C

@@ -527,6 +542,12 @@ MooseApp::setupOptions()
      _syntax.addDependency("mesh_only", "setup_mesh_complete");
      _action_warehouse.setFinalTask("mesh_only");
    }
+    else if (getParam<std::string>("split_mesh") != "")


Does isParamValid() not work here and in other usages of this new parameter?

permcody · 2018-02-01T15:25:13Z

framework/include/utils/MooseUtils.h

+ * Removes any file extension from the fiven string s (i.e. any ".[extension]" suffix of s) and
+ * returns the result.
+ */
+std::string stripExtension(const std::string & s);


This is necessary because checkpoint files will be in different locations and named differently depending on mesh splitting and processor number. The naive file check we did before won't work. So for now we skip the check for cpr/cpa files. This is necessary for an upcoming libmesh PR that is necessary for the upcoming idaholab#10623 moose PR.

rwcarlsen · 2018-02-12T19:38:54Z

This should be ready for another look now that the new libmesh stuff is in.

permcody · 2018-02-12T21:43:32Z

framework/src/actions/SetupMeshAction.C

+      new_pars.set<MeshFileName>("file") = split_file;
+      new_pars.set<MooseApp *>("_moose_app") = _moose_object_pars.get<MooseApp *>("_moose_app");
+      new_pars.set<MooseEnum>("parallel_type") = _moose_object_pars.get<MooseEnum>("parallel_type");
+      new_pars.set<MooseEnum>("parallel_type") = "distributed";


I don't think this is correct, two sets in a row for the same parameter?

permcody · 2018-02-12T21:45:48Z

framework/src/actions/SetupMeshAction.C

+    {
+      if (split_file == "")
+        mooseError("cannot use split mesh with non-file mesh without specifying --split-file on "
+                   "command line");


I guess this is OK, it would be cool if we could use the virtual function but that might be a little tricky before the object is created.

For generated mesh, we don't know the name of the pre-split files. There is no way we could divine it. The autosplit generated mesh and run in one go is a separate thing that will have to be implemented later.

I'm with @rwcarlsen here - this is the right thing to do.

permcody · 2018-02-12T21:46:36Z

framework/src/actions/SetupMeshAction.C

@@ -152,6 +153,34 @@ SetupMeshAction::setupMesh(MooseMesh * mesh)
 void
 SetupMeshAction::act()
 {
+  // switch non-file meshes to be a file-mesh if using a pre-split mesh configuration.
+  if (_app.parameters().get<bool>("use_split"))


You shouldn't need to run this logic twice (or more). You only need to run it during the "setup_mesh" task.

Not sure what you are saying here. Could you clarify?

Take a look at the code below this if statement (e.g. if _current_task == ...). This Action gets fired multiple time to perform different tasks. You only need to run this new logic for the setup task, not the init task.

To be clear, it's not about efficiency or anything, just getting it right logically so we don't look back at this later and wonder why this block runs for every task.

Ah - I see - good catch. It is easy to forget that multiple tasks can use/run the same action.

rwcarlsen · 2018-02-13T20:22:04Z

Just waiting for a review update from @friedmud.

friedmud · 2018-02-14T20:27:43Z

Grabbing this now...

Fixes idaholab#10458. Note that the mesh splitting won't work right until we get another libmesh update in :-(

friedmud

Need to add .md documentation on how to use this.

friedmud · 2018-02-14T22:29:37Z

When you didn't specify --split-file you get an error from every processor:

mpiexec -n 4 ../../../moose_test-opt -i simple_diffusion.i  --split-mesh "2,4"





*** ERROR ***
Output mesh file name must be specified (with --split-file) when splitting non-file-based meshes






*** ERROR ***
Output mesh file name must be specified (with --split-file) when splitting non-file-based meshes






*** ERROR ***
Output mesh file name must be specified (with --split-file) when splitting non-file-based meshes


application called MPI_Abort(MPI_COMM_WORLD, 1) - process 2




*** ERROR ***
Output mesh file name must be specified (with --split-file) when splitting non-file-based meshes


application called MPI_Abort(MPI_COMM_WORLD, 1) - process 3
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1

If you're running on thousands of procs that is going to suck. Let's just output that error on proc 0.

friedmud · 2018-02-14T22:32:41Z

When we specify a filename using --split-file should we append .cpr on the name of it? It's a little funky to just get a directory out named "stuff" or whatever... but it could be ok

permcody · 2018-02-14T22:35:16Z

So I've often thought about what to do about parallel mooseErrors. It's actually a really big pain because we have mooseErrors sprinkled all over the code and more often than not, they are in common code paths so this probably isn't isolated. How many times have you submitted an input file with a simple problem only to get a thousand lines of mooseErrors from every rank?

Outputting on processor zero is actually reasonable when we know it's on a common code path. Maybe we can give this some thought and be smart about when errors come out? I guess for now for this very common and specific error we can add one more if (processor_id() == 0) branch, but we should think about making errors more useful as we continue to evolve MOOSE.

friedmud · 2018-02-14T22:35:47Z

Actually: we should reject the filename if you don't put .cpr on the end of it!

It turns out we can't read it!

[gastdr][~/projects/moose/test/tests/kernels/simple_diffusion]> mpiexec -n 4 ../../../moose_test-opt -i simple_diffusion.i  --split-file junk --use-split

 ERROR: Unrecognized file extension: junk
   I understand the following:

     *.e    -- Sandia's ExodusII format
     *.exd  -- Sandia's ExodusII format
     *.gmv  -- LANL's General Mesh Viewer format
     *.mat  -- Matlab triangular ASCII file
     *.n    -- Sandia's Nemesis format
     *.nem  -- Sandia's Nemesis format
     *.off  -- OOGL OFF surface format
     *.ucd  -- AVS's ASCII UCD format
     *.unv  -- I-deas Universal format
     *.vtu  -- Paraview VTK format
     *.inp  -- Abaqus .inp format
     *.xda  -- libMesh ASCII format
     *.xdr  -- libMesh binary format
     *.gz   -- any above format gzipped
     *.bz2  -- any above format bzip2'ed
     *.xz   -- any above format xzipped
     *.cpa  -- libMesh Checkpoint ASCII format
     *.cpr  -- libMesh Checkpoint binary format

[0] src/mesh/namebased_io.C, line 261, compiled nodate at notime




*** ERROR ***
 ERROR: Unrecognized file extension: junk
   I understand the following:

     *.e    -- Sandia's ExodusII format
     *.exd  -- Sandia's ExodusII format
     *.gmv  -- LANL's General Mesh Viewer format
     *.mat  -- Matlab triangular ASCII file
     *.n    -- Sandia's Nemesis format
     *.nem  -- Sandia's Nemesis format
     *.off  -- OOGL OFF surface format
     *.ucd  -- AVS's ASCII UCD format
     *.unv  -- I-deas Universal format
     *.vtu  -- Paraview VTK format
     *.inp  -- Abaqus .inp format
     *.xda  -- libMesh ASCII format
     *.xdr  -- libMesh binary format
     *.gz   -- any above format gzipped
     *.bz2  -- any above format bzip2'ed
     *.xz   -- any above format xzipped
     *.cpa  -- libMesh Checkpoint ASCII format
     *.cpr  -- libMesh Checkpoint binary format



application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0

friedmud · 2018-02-14T22:36:46Z

@permcody what about just having a mooseErrorOnProcessor0() macro?

friedmud · 2018-02-14T22:37:03Z

Or - I guess, just an argument to mooseError()

permcody · 2018-02-14T22:51:27Z

The problem with that is that it relies on the developer using it correctly which might not always be as easy as it seems. You run the risk of flying right through an error condition if for some reason processor zero doesn't encounter the error but some of the other ranks do, but are told to just carry on.

I can think of two other alternatives that might be better (might not). We could use the libMesh "parallel_only" macro that does an assert to make sure the error is in a parallel region. The problem with that, is the assert goes away in your big parallel run when you actually need it to error and you still miss the error in the worst case.

Another alternative would be to still call abort on the non-zero ranks but not print? The downside to that is that on a really big job you run the risk of just terminating with no error message what-so-ever which would probably be way worse than seeing a thousand of the same message.

I'm not sure I like any alternative.

friedmud · 2018-02-14T22:53:41Z

framework/src/actions/SplitMeshAction.C

+    cpr->binary() = true;
+    auto fname = MooseUtils::stripExtension(mesh->getFileName()) + ".cpr";
+    if (mesh->getFileName() == "")
+      fname = _app.parameters().get<std::string>("split_file");


There are a few ways to go here. Line 52 could be before line 50... but that would mean that running the same command-line twice might not work.

Probably better just to error if the split_file does not have a .cpr.

Actually - let's add the .cpr here... and then make sure it gets added properly when we're reading in too.

It is possible that it is cpa or the user renamed it to something random though. I think if the user uses the --split-file flag we should just use that wholesale - this especially makes sense with tab completion.

It actually doesn't work though - have you tried it?

If the user names it anything other than .cpr it doesn't work. I think we should make sure that .cpr is on the end (if the user passes it just leave it alone - if they don't then append it... for both writing and reading)

I'll add the check for the extension when writing.

permcody · 2018-02-14T22:53:47Z

Option #3: Just get in the habit of using separate file outputs for big jobs or ignoring your error stream instead of combining it into your output stream. This is what I've done recently because I almost never want to see errors.

friedmud · 2018-02-14T23:00:21Z

@permcody : well I guess it's not something to solve here

rwcarlsen · 2018-02-15T22:58:51Z

Recommendations for where (what file(s)) to put docs? This isn't really a system/object so it can't really follow the "docs go in the file corresponding to this class/object" pattern.

friedmud · 2018-02-19T16:11:53Z

Add a short section here: http://mooseframework.org/moose/documentation/systems/Mesh/index.html

Then link to a longer explanation.

rwcarlsen · 2018-02-19T17:39:23Z

Alright - I pushed up some docs.

friedmud

Documentation is 👍

But we need to get the name extension thing right...

friedmud · 2018-02-19T20:08:24Z

framework/src/actions/SplitMeshAction.C

+    cpr->binary() = true;
+    auto fname = MooseUtils::stripExtension(mesh->getFileName()) + ".cpr";
+    if (mesh->getFileName() == "")
+      fname = _app.parameters().get<std::string>("split_file");


It actually doesn't work though - have you tried it?

If the user names it anything other than .cpr it doesn't work. I think we should make sure that .cpr is on the end (if the user passes it just leave it alone - if they don't then append it... for both writing and reading)

friedmud

You need to add the .cpr for both writing and reading or you lose symmetry in the commands:

[gastdr][~/projects/moose/test/tests/kernels/simple_diffusion]> ../../../moose_test-opt -i simple_diffusion.i --split-mesh 4 --split-file junk

Splitting 4 ways...
    - writing 4 files per process...

[gastdr][~/projects/moose/test/tests/kernels/simple_diffusion]> ls
gold			junk.cpr		simple_diffusion.i	simple_diffusion_out.e	speedtests		tests
[gastdr][~/projects/moose/test/tests/kernels/simple_diffusion]> mpiexec -n 4 ../../../moose_test-opt -i simple_diffusion.i --use-split --split-file junk

*** ERROR ***
cannot locate mesh file 'junk'

rwcarlsen · 2018-02-20T16:54:50Z

Done.

friedmud · 2018-02-20T17:43:42Z

I'll merge this when the testing is complete!

Thanks for working on this... it's really slick!

follow up to idaholab#10623

permcody requested changes Jan 26, 2018

View reviewed changes

friedmud self-assigned this Jan 26, 2018

friedmud requested changes Jan 26, 2018

View reviewed changes

permcody added the PR: Address comments label Jan 29, 2018

rwcarlsen force-pushed the split-mesh branch from af78a99 to dd8fb93 Compare January 31, 2018 20:04

moosebuild removed the PR: Address comments label Jan 31, 2018

permcody reviewed Feb 1, 2018

View reviewed changes

rwcarlsen force-pushed the split-mesh branch 2 times, most recently from 000ecc5 to f46b090 Compare February 2, 2018 19:40

rwcarlsen force-pushed the split-mesh branch 2 times, most recently from 82279e7 to 874c694 Compare February 12, 2018 19:37

permcody requested changes Feb 12, 2018

View reviewed changes

rwcarlsen force-pushed the split-mesh branch from 874c694 to db412d7 Compare February 13, 2018 17:00

permcody added PR: Address comments and removed PR: Address comments labels Feb 13, 2018

permcody previously approved these changes Feb 13, 2018

View reviewed changes

rwcarlsen dismissed permcody’s stale review via b0184a9 February 14, 2018 22:13

rwcarlsen force-pushed the split-mesh branch 2 times, most recently from b0184a9 to f4f20de Compare February 14, 2018 22:15

implement convenience workflow for using distributed mesh

add095b

Fixes idaholab#10458. Note that the mesh splitting won't work right until we get another libmesh update in :-(

rwcarlsen force-pushed the split-mesh branch from f4f20de to add095b Compare February 14, 2018 22:17

friedmud requested changes Feb 14, 2018

View reviewed changes

friedmud reviewed Feb 14, 2018

View reviewed changes

friedmud added the PR: Address comments label Feb 14, 2018

don't print errors on all ranks

16daf88

moosebuild removed the PR: Address comments label Feb 15, 2018

add usage docs

6577952

friedmud requested changes Feb 19, 2018

View reviewed changes

friedmud requested changes Feb 20, 2018

View reviewed changes

always add cpr extension if missing

59b8479

rwcarlsen force-pushed the split-mesh branch from d802689 to 59b8479 Compare February 20, 2018 16:53

friedmud approved these changes Feb 20, 2018

View reviewed changes

friedmud merged commit 94266fa into idaholab:devel Feb 20, 2018

rwcarlsen added a commit to rwcarlsen/moose that referenced this pull request Feb 21, 2018

add mesh splitting tests

1ac5a31

follow up to idaholab#10623

rwcarlsen mentioned this pull request Feb 21, 2018

add mesh splitting tests #10849

Merged

rwcarlsen added a commit to rwcarlsen/moose that referenced this pull request Feb 21, 2018

add mesh splitting tests

7a6c86c

follow up to idaholab#10623

rwcarlsen added a commit to rwcarlsen/moose that referenced this pull request Feb 21, 2018

add mesh splitting tests

8e79d7b

follow up to idaholab#10623

rwcarlsen added a commit to rwcarlsen/moose that referenced this pull request Feb 21, 2018

add mesh splitting tests

2f48a03

follow up to idaholab#10623

rwcarlsen deleted the split-mesh branch March 12, 2018 15:46

permcody mentioned this pull request Jul 2, 2018

Better Pre-Split Naming #10088

Closed

implement convenience workflow for using distributed mesh #10623

implement convenience workflow for using distributed mesh #10623

Conversation

rwcarlsen commented Jan 26, 2018

moosebuild commented Jan 26, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rwcarlsen Jan 26, 2018 • edited Loading

Choose a reason for hiding this comment

permcody Jan 26, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

friedmud left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

permcody left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rwcarlsen commented Feb 12, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rwcarlsen commented Feb 13, 2018

friedmud commented Feb 14, 2018

friedmud left a comment

Choose a reason for hiding this comment

friedmud commented Feb 14, 2018

friedmud commented Feb 14, 2018

permcody commented Feb 14, 2018

friedmud commented Feb 14, 2018

friedmud commented Feb 14, 2018

friedmud commented Feb 14, 2018

permcody commented Feb 14, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

permcody commented Feb 14, 2018

friedmud commented Feb 14, 2018

rwcarlsen commented Feb 15, 2018

friedmud commented Feb 19, 2018

rwcarlsen commented Feb 19, 2018

friedmud left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

moosebuild commented Jan 26, 2018 •

edited

Loading

rwcarlsen Jan 26, 2018 •

edited

Loading

permcody Jan 26, 2018 •

edited

Loading

permcody commented Feb 14, 2018 •

edited

Loading

friedmud left a comment •

edited

Loading