[API] TensorFlow Graph Transformer #39

phi-dbq · 2017-08-08T21:28:59Z

What changes are proposed in this pull request?

API design and reference implementation of a Spark MLlib Transformer from any TensorFlow Graph.

Notice
This implementation is only a proof-of-concept reference. We will break it up into multiple PRs with finer grain tests and documentations.

This is not an implementation of the (attention-based) Transformer architecture (https://arxiv.org/abs/1706.03762).

How is this patch tested?

Integration tests
Manual tests

phi-dbq · 2017-08-08T21:35:07Z

python/sparkdl/transformers/tf_tensor.py

+              since a new session is created inside this transformer.
+    """
+
+    graphFunction = Param(Params._dummy(), "graphFunction",


The intention is to provide this as the Transformer API and provide utilities to convert any TensorFlow/Keras graph (or checkpoints) into a GraphFunction object.

GraphFunction and IsolatedSession were added as internal classes not to be exposed to users, only to be used within the internals of sparkdl, so we shouldn't have GraphFunction be a param user defines. These are foreign concepts to TensorFlow users. We should stick with what users know as much as possible so that this functionality is as easy for them to use as possible. Making this API match the one TFoS guys are building (https://github.com/yahoo/TensorFlowOnSpark/pull/114/files) would be better. If you think the proposed API in https://github.com/yahoo/TensorFlowOnSpark/pull/114/files is not as good, we should discuss.

We may not need all the params the other API has, but if that is the case, we should have this API start with params that are a strict subset of the other API.

codecov-io · 2017-08-09T00:40:17Z

Codecov Report

Merging #39 into master will increase coverage by 1.81%.
The diff coverage is 92.06%.

@@            Coverage Diff             @@
##           master      #39      +/-   ##
==========================================
+ Coverage   82.82%   84.64%   +1.81%     
==========================================
  Files          23       26       +3     
  Lines        1217     1485     +268     
  Branches        5        5              
==========================================
+ Hits         1008     1257     +249     
- Misses        209      228      +19

Impacted Files	Coverage Δ
python/sparkdl/param/__init__.py	`100% <100%> (ø)`	⬆️
python/sparkdl/graph/builder.py	`93.75% <100%> (+0.05%)`	⬆️
python/sparkdl/transformers/tf_tensor.py	`100% <100%> (ø)`
python/sparkdl/__init__.py	`100% <100%> (ø)`	⬆️
python/sparkdl/graph/utils.py	`89.01% <72.72%> (-6.06%)`	⬇️
python/sparkdl/param/converters.py	`80.3% <80.3%> (ø)`
python/sparkdl/param/shared_params.py	`81.25% <84.37%> (+1.04%)`	⬆️
python/sparkdl/graph/input.py	`99.29% <99.29%> (ø)`
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3f668d9...b232b3c. Read the comment docs.

sueann

Left some high-level comments - we can discuss in person if you don't agree with them.

sueann · 2017-08-09T18:14:05Z

python/sparkdl/transformers/tf_tensor.py

+              since a new session is created inside this transformer.
+    """
+
+    graphFunction = Param(Params._dummy(), "graphFunction",


GraphFunction and IsolatedSession were added as internal classes not to be exposed to users, only to be used within the internals of sparkdl, so we shouldn't have GraphFunction be a param user defines. These are foreign concepts to TensorFlow users. We should stick with what users know as much as possible so that this functionality is as easy for them to use as possible. Making this API match the one TFoS guys are building (https://github.com/yahoo/TensorFlowOnSpark/pull/114/files) would be better. If you think the proposed API in https://github.com/yahoo/TensorFlowOnSpark/pull/114/files is not as good, we should discuss.

sueann · 2017-08-09T18:14:39Z

python/sparkdl/transformers/tf_tensor.py

+                self.getInputCol(), orig_in_name).select(orig_in_name)
+            output_df = tfs.map_blocks(fetches, input_df)
+            orig_out_name = tfx.op_name(issn.graph, fetches[0])
+            final_df = output_df.withColumn(


you can do a rename for the column instead

sueann · 2017-08-09T18:19:11Z

python/sparkdl/transformers/tf_tensor.py

+              since a new session is created inside this transformer.
+    """
+
+    graphFunction = Param(Params._dummy(), "graphFunction",


We may not need all the params the other API has, but if that is the case, we should have this API start with params that are a strict subset of the other API.

phi-dbq · 2017-08-09T22:28:31Z

I am designing the API around TensorFlow Graph + Input/Output Tensors.
The TFoS API takes a checkpoint directory for the (meta) graph. We can get a utility function to convert it to a TensorFlow Graph.

phi-dbq · 2017-08-10T00:33:03Z

python/sparkdl/transformers/param.py

 # New in sparkdl

 class SparkDLTypeConverters(object):

    @staticmethod
+    def toTFGraph(value):


Directly importing tf.GraphDef without the input tensor mapping is not working.
Thus the user either provides a TensorFlow Graph or provide a GraphFunction (for developers).

sueann

left some more comments. Let's sync offline about the API.

sueann · 2017-08-10T19:19:49Z

python/sparkdl/transformers/param.py

@@ -45,6 +46,8 @@ def wrapper(self, *args, **kwargs):
    return wrapper


+""" TensorFlow Specific Parameters """


hasInputCol is not TensorFlow specific. in fact, it was copied from pyspark...!

sueann · 2017-08-10T19:26:30Z

python/sparkdl/transformers/tf_tensor.py

+        assert len(feeds) == 1, 'only support single input TF model'
+        assert len(fetches) == 1, 'only support single output TF model'
+
+        # Change the column name for TensorFrames


it's not much code -- why not just put it in the transformer and skip having to convert from/to GraphFunction? also can keep using the IsolatedSession there then.

sueann · 2017-08-10T19:27:31Z

python/sparkdl/transformers/tf_tensor.py

+
+    return final_df
+
+class TFOneDimTensorTransformer(Transformer, HasInputCol, HasOutputCol,


I'd just name it TFTransformer or TFModel. We can add to the doc that it only supports 1-d arrays currently.

sueann · 2017-08-10T19:32:04Z

python/tests/transformers/tf_tensor_test.py

+        self.assertTrue(np.allclose(out_ref, out_tgt))
+
+
+    def test_simple_graph_function(self):


what does this test? it seems like the difference between this and test_simple() is that this one uses GraphFunction and IsolatedSession? I don't think that test belongs in this class (and the functionality should have been tested in the Suites for GraphFunction/IsolatedSession anyway).

anfeng · 2017-08-11T00:15:27Z

python/sparkdl/transformers/param.py

+        return self.getOrDefault(self.outputTensor)
+
+
+class HasInputTensor(Params):


Normally, we need to handle multiple 1+ input tensors. See TensorFlow Serving as a reference.

Also for output tensors. See https://github.com/tensorflow/serving/blob/master/tensorflow_serving/example/mnist_saved_model.py#L109-L114

anfeng · 2017-08-11T00:17:45Z

python/sparkdl/transformers/tf_tensor.py

+
+    @keyword_only
+    def setParams(self, tfGraph=None, inputTensor=None, outputTensor=None,
+                  inputCol=None, outputCol=None):


We need to handle multiple input colmns.

Also for output columns

I am wondering if we want the tensors to be in separate columns or in fields of a nested column?

sueann · 2017-08-11T23:15:47Z

python/sparkdl/transformers/tf_tensor.py

+
+logger = logging.getLogger('sparkdl')
+
+class TFTensorTransformer(Transformer, HasTFGraph, HasTFHParams, HasInputMapping, HasOutputMapping):


TFTensorTransformer is redundant since it spells out to be TensorFlowTensorTransformer. Why not just TFTransformer or TFModel?

sueann · 2017-08-11T23:19:14Z

python/sparkdl/transformers/tf_tensor.py

+        analyzed_df = tfs.analyze(df)
+
+        with IsolatedSession() as issn:
+            _, fetches = issn.importGraphFunction(gfn, prefix='')


Lines 83-33: why do you need to create a GraphFunction and then importGraphFunction to get fetches vs just using the fetches from above?

Somewhat related: analyzed_df = tfs.analyze(df) (line 85) doesn't have to be called right there, right? Could be called before the first IsolatedSession here?

The use of two sessions is due to the fact that exporting to GraphFunction modifies the graph and it has to be imported in another session.

oh i see... if we just need strip_and_freeze we should just call that here directly because this is hiding the fact that that is what (and all) we are doing.

It also seems that importing graph_def directly is a bit flaky. importGraphFunction attaches the names and tensors correctly. It might just save us a few lines of code.

@phi-dbq what do you mean by flaky? I do not understand why simply using import_graph_def would not work here.

Sometimes one might use a session without resetting the default graph.

with tf.Session() as sess: tf.import_graph_def(graph_def, name='') sess.run(...)

In case there is a default graph and there is a name conflict, the tensors in the imported graph will be renamed automatically.
(if we pass an exising graph as tf.Session(grarph=existing_graph) as sess, it seems to use the default graph properly inside the session. )

But in this case there isn't another graph in the session right? So there is no need to have the complex logic around the graph / graphdef. Could lines 70-92 just be:

analyzed_df = tfs.analyze(df) # prune the graph for tensorframes with tf.session(graph=self.getTFGraph()) as sess: fetches = [tfx.get_tensor(sess.graph, tnsr) for tnsr, output_colname in self.getOutputMapping()] gdef = tfx.strip_and_freeze_until(fetches, sess.graph, sess) with tf.session() as sess2: output_names = [tfx.validated_output(sess.graph, elem) for elem in fetches] tf.import_graph_def(gdef, input_map=None, return_elements=output_names, name='') final_fetches = [tfx.get_tensor(sess2.graph, name) for name in output_names] out_df = tfs.map_blocks(final_fetches, analyzed_df) for tnsr, output_colname in self.getOutputMapping(): out_df = out_df.withColumnRenamed(tfx.op_name(sess.graph, tnsr), output_colname)

?

With IsolatedSession and GraphFunction, it's hard for anyone not intimately familiar with those classes to understand what's going on here. It makes it seem like you are just creating a GraphFunction object then getting back exactly what you put in. With more explicit code like above, it's easier to tell what exactly we're doing (pruning the graph into graphdef and re-importing). Also it seems shorter...

A few questions:
1/ I thought the tensorframes changes were done to remove the necessity for freezing+reimporting? But I may be wrong there (didn't really look at the tensorframes PR).
2/ In https://github.com/databricks/spark-deep-learning/blob/master/python/sparkdl/graph/builder.py#L91 (and potentially in the code I wrote above), do we need to check that no output has been stripped from the graph? Or is that not possible because the outputs were fed into the strip/freeze function and would be kept even if an output was "floating" and would have been stripped normally?

phi-dbq · 2017-08-12T00:16:36Z

python/tests/transformers/tf_tensor_test.py

+        # Using the Transformer
+        transformer = TFModelTransformer(tfGraph=gfn,
+                                         inputMapping={
+                                             input_col: gfn.input_names[0]


We can either provide names or tensors for the mappings

phi-dbq · 2017-08-12T03:11:20Z

python/tests/transformers/tf_tensor_test.py

+        # We want to clear device assignment in order to run it anywhere we want
+        with IsolatedSession() as issn:          
+            saver = tf.train.import_meta_graph('{}.meta'.format(saved_path), clear_devices=True)
+            saver.restore(issn.sess, saved_path)


We have to export this checkpoint as GraphFunction in the same session where it is restored. Since the variables are only initialized in this session.

Also it seems that we have to import the meta_graph first or there won't be any graph in this session (tried with either IsolatedSession or standard tf.Session).

phi-dbq · 2017-08-14T16:54:33Z

python/tests/transformers/tf_tensor_test.py

+            z = tf.reduce_mean(x * w, axis=1, name='tnsrOut')            
+            sess.run(w.initializer)
+            saver = tf.train.Saver(var_list=[w])
+            saved_path = saver.save(sess, ckpt_dir, global_step=2702)


Save the model and record the current global_step. Then we'd expect files like these.

model.ckpt-2702.index model.ckpt-2702.meta model.ckpt-2702.data-00000-of-00001

phi-dbq · 2017-08-18T17:22:52Z

python/tests/transformers/tf_tensor_test.py

+                [tfx.get_tensor(issn.graph, 'tnsrOut')])
+
+        transformer = TFModelTransformer(tfGraph=gfn,
+                                         inputMapping={


We should be better reversing the input mapping to match feed_dict in tf.session.run and TensorFrames map_row.

thunterdb

I have not finished the review, but I have a couple of comments already. I will have to finish on Monday.

thunterdb · 2017-08-17T20:34:02Z

python/sparkdl/transformers/tf_tensor.py

+            for colname, tnsr in self.getInputMapping().items():
+                feeds.append(tfx.get_tensor(issn.graph, tnsr))
+                new_colname = tfx.op_name(issn.graph, tnsr)
+                df = df.withColumnRenamed(colname, new_colname)


this may not always work because there may be conflicts with existing columns, but this is rare enough I think that we do not need to handle it currently.

thunterdb · 2017-08-17T20:38:37Z

python/sparkdl/transformers/param.py

+
+# New in sparkdl
+
+class HasOutputMapping(Params):


This class is too complicated: I do not understand what is inside that parameter (actually, I do, but I had to reverse-engineer the logic to figure this out). Here is what we should do: store a Seq[(String, String)] for that parameter so that the order is clear, and only strings. Right now, you accept all sorts of objects such as tensors, which makes it hard to reason about. Most important, document that what each element means in the document. At least as a code comment.

The reason for doing that is that you convert to primitive types, and that the type of the param is much more clear.

thunterdb · 2017-08-17T20:38:40Z

python/sparkdl/transformers/param.py

+        super(HasOutputMapping, self).__init__()
+
+    def setOutputMapping(self, value):
+        return self._set(outputMapping=value)


Here, instead of being as simple, do some conversions: dicts should be converted to lists, sorted by key order; tensors should be converted to the string names, etc.

thunterdb · 2017-08-17T20:39:16Z

python/sparkdl/transformers/param.py

+    def setOutputMapping(self, value):
+        return self._set(outputMapping=value)
+
+    def getOutputMapping(self):


Says what it returns (list of tensor name -> output name)

thunterdb · 2017-08-17T20:40:02Z

python/sparkdl/transformers/param.py

+
+    def setOutputMapping(self, value):
+        return self._set(outputMapping=value)
+


It would be nice to check that the tensors are part of the graph, but this is probably too fancy given the code structure.

thunterdb · 2017-08-17T20:40:29Z

python/sparkdl/transformers/param.py

+            raise TypeError("Could not convert %s to TensorFlow Graph" % type(value))
+
+    @staticmethod
+    def asColumnToTensorMap(value):


based on my comment below, you do not need that function. It can be written just in a few lines during the conversion.

Also, instead of making something static and publick, just make a small helper funciton that is not exported.

I am using a converter function in sparkdl.graph.utils to convert the tensor/operation object into the corresponding operation name. If the operation is named inputPlaceholder, the tensor name is inputPlaceholder:0. TensorFrames expects the operation name.

thunterdb · 2017-08-17T20:41:19Z

python/sparkdl/transformers/param.py

+        raise TypeError("Could not convert %s to TensorFlow Tensor" % type(value))
+
+    @staticmethod
+    def asTensorToColumnMap(value):


you do not need this separate function at all, based on my quick analysis. Just one is enough.

I take it back, you need each of the function (input and output), but instead of reverting the dict, you can simply have have a conversion function that you use like [k, convert(tensor) for ...] and [convert(tensor), name for ...]

I am using a converter function in sparkdl.graph.utils to convert the tensor/operation object into the corresponding operation name. If the operation is named inputPlaceholder, the tensor name is inputPlaceholder:0. TensorFrames expects the operation name.

thunterdb · 2017-08-17T20:45:09Z

python/sparkdl/graph/utils.py

@@ -165,6 +165,17 @@ def validated_input(graph, tfobj_or_name):
        ('input must be Placeholder, but get', op.type)
    return name

+def is_interpretable_tensor_type(tfobj_or_name):


Let's not have such a function.

Here is what you are currently doing:

checking that something belongs to a union of types

storing it into a structure

then using it for lookup, which uses different branches depending on the type and does some conversions.
We can drastically simplify that:

convert the something to just a string

store this simple type (string)

do a simpler lookup using this simple type
It is easier to reason about the flow of info.

thunterdb · 2017-08-17T20:47:30Z

python/sparkdl/transformers/param.py

+
+
+class HasInputMapping(Params):
+    inputMapping = Param(Params._dummy(), "inputMapping",


Same comment as above.

thunterdb · 2017-08-17T20:50:01Z

python/sparkdl/transformers/param.py

+
+
+class HasTFGraph(Params):
+    tfGraph = Param(Params._dummy(), "tfGraph",


Doc please.

thunterdb

@phi-dbq I have a few comments. I think the core function of the code will be greatly simplified once I do a tiny change to tensorframes.

thunterdb · 2017-08-22T17:19:43Z

python/sparkdl/graph/utils.py

+        return name
+    elif hasattr(tfobj_or_name, 'graph'):
+        tfobj = tfobj_or_name
+        return get_tensor(tfobj.graph, tfobj).name


tensor_name

thunterdb · 2017-08-22T17:20:41Z

python/sparkdl/graph/utils.py


-def as_op_name(name):
+def as_op_name(tfobj_or_name):


We have 5 functions that compute some names in this file, this is at least two too many.

thunterdb · 2017-08-22T17:47:38Z

python/sparkdl/transformers/tf_tensor.py

+
+            gfn = issn.asGraphFunction(feeds, fetches, strip_and_freeze=True)
+
+        analyzed_df = tfs.analyze(df)


The analyze() function is not smart enough currently to know that the data is already analyzed, so it make take a substantial amount of time. This is an easy fix in tensorframes though.

thunterdb · 2017-08-22T17:48:55Z

python/sparkdl/transformers/tf_tensor.py

+        df = dataset
+        output_renaming = {}
+
+        with IsolatedSession(graph=self.getTFGraph()) as issn:


I see why you need this currently. There is a feed_dict arguments for map_rows that does exactly what you want in tensorframes. It still needs to be implemented for map_blocks, but this is a quick fix.

Once this argument is there, this block of code will not be required anymore.

Also, errors here are going to be confusing because you are renaming the columns.

thunterdb · 2017-08-22T17:53:48Z

python/sparkdl/transformers/tf_tensor.py

+        analyzed_df = tfs.analyze(df)
+
+        with IsolatedSession() as issn:
+            _, fetches = issn.importGraphFunction(gfn, prefix='')


@phi-dbq what do you mean by flaky? I do not understand why simply using import_graph_def would not work here.

thunterdb · 2017-08-22T17:55:19Z

python/tests/transformers/tf_tensor_test.py

+        # Apply the transform
+        transfomer = TFTransformer(tfGraph=graph,
+                                   inputMapping={
+                                       'vec': x


why are there new lines here?

sueann · 2017-08-23T20:40:03Z

python/sparkdl/transformers/param.py

-        # TODO: we may want to support tf.GraphDef in the future instead of tf.Graph since user
-        # is less likely to mess up using GraphDef vs Graph (e.g. constants vs variables).
-        if isinstance(value, tf.Graph):
+    def toGraphFunction(value):


this is unused?

sueann · 2017-08-23T20:40:25Z

python/sparkdl/transformers/param.py

@@ -92,11 +123,69 @@ def getOutputCol(self):
        return self.getOrDefault(self.outputCol)


-# New in sparkdl
+class HasOutputCols(Params):


sueann · 2017-09-13T22:52:37Z

@phi-dbq, left you a few more comments skimming the latest but I'm not sure if I'm supposed to review again yet. Let me know when it's ready.

phi-dbq · 2017-10-17T19:15:48Z

Closing this PR in favor of the stacked PR holistic view
phi-dbq#12

phi-dbq commented Aug 8, 2017

View reviewed changes

phi-dbq force-pushed the tf-1d-transformer branch from 0f083c4 to b731e55 Compare August 8, 2017 21:41

phi-dbq requested a review from sueann August 8, 2017 21:42

phi-dbq force-pushed the tf-1d-transformer branch from 5108164 to 0c90493 Compare August 9, 2017 18:03

sueann requested changes Aug 9, 2017

View reviewed changes

phi-dbq force-pushed the tf-1d-transformer branch from 2039cd5 to 8e33759 Compare August 9, 2017 22:26

phi-dbq requested a review from sueann August 9, 2017 22:26

phi-dbq commented Aug 10, 2017

View reviewed changes

sueann reviewed Aug 10, 2017

View reviewed changes

anfeng reviewed Aug 11, 2017

View reviewed changes

sueann reviewed Aug 11, 2017

View reviewed changes

phi-dbq force-pushed the tf-1d-transformer branch 2 times, most recently from 1365723 to bee8d18 Compare August 11, 2017 23:35

phi-dbq commented Aug 12, 2017

View reviewed changes

phi-dbq force-pushed the tf-1d-transformer branch from 976ac08 to bab6724 Compare August 12, 2017 03:08

phi-dbq commented Aug 12, 2017

View reviewed changes

phi-dbq commented Aug 14, 2017

View reviewed changes

phi-dbq commented Aug 18, 2017

View reviewed changes

thunterdb reviewed Aug 18, 2017

View reviewed changes

phi-dbq force-pushed the tf-1d-transformer branch 4 times, most recently from c4b6c15 to ad481a7 Compare August 21, 2017 22:16

thunterdb reviewed Aug 22, 2017

View reviewed changes

sueann reviewed Aug 23, 2017

View reviewed changes

phi-dbq force-pushed the tf-1d-transformer branch 4 times, most recently from 80795d7 to ee9b9e2 Compare September 14, 2017 02:53

test refactor

9b3fe86

phi-dbq force-pushed the tf-1d-transformer branch from ee9b9e2 to 9b3fe86 Compare September 14, 2017 02:56

phi-dbq added 9 commits September 15, 2017 19:58

Merge remote-tracking branch 'upstream/master' into tf-1d-transformer

8c32501

(wip) consolidating params

dbd9aaa

rebase upstream

4572205

import params fix

1cc7591

(wip) TFInputGraph impl

2fc6787

(wip) moving to new API

889df0a

(wip) enable saved_model tests

86cd6d9

(wip) enable checkpoint test

ac09182

(wip) enable multiple tensor tests

6b22eed

phi-dbq force-pushed the tf-1d-transformer branch from 196244e to 111c427 Compare September 17, 2017 03:25

enable all tests

a3517d6

phi-dbq force-pushed the tf-1d-transformer branch from 111c427 to a3517d6 Compare September 17, 2017 03:27

params and converters

457a4c2

phi-dbq mentioned this pull request Sep 18, 2017

TensorFlow Graph Transformer Part-1: Params and Converters #49

Merged

phi-dbq added 2 commits September 18, 2017 15:27

tests

323939a

Merge branch 'tf-transformer-part1' into api-tf-transformer

6e46073

phi-dbq force-pushed the tf-1d-transformer branch 2 times, most recently from 439df5b to 2d25c32 Compare September 19, 2017 01:01

optimize graph for inference

b232b3c

phi-dbq force-pushed the tf-1d-transformer branch from 2d25c32 to b232b3c Compare September 19, 2017 03:47

phi-dbq closed this Oct 17, 2017

phi-dbq mentioned this pull request Nov 17, 2017

TFTransformer Part-3 Test Refactor phi-dbq/spark-deep-learning#14

Merged

1 task

		@@ -45,6 +46,8 @@ def wrapper(self, args, *kwargs):
		return wrapper


		""" TensorFlow Specific Parameters """


		return final_df

		class TFOneDimTensorTransformer(Transformer, HasInputCol, HasOutputCol,

		self.assertTrue(np.allclose(out_ref, out_tgt))


		def test_simple_graph_function(self):

		return self.getOrDefault(self.outputTensor)


		class HasInputTensor(Params):


		logger = logging.getLogger('sparkdl')

		class TFTensorTransformer(Transformer, HasTFGraph, HasTFHParams, HasInputMapping, HasOutputMapping):


		def setOutputMapping(self, value):
		return self._set(outputMapping=value)



		class HasInputMapping(Params):
		inputMapping = Param(Params._dummy(), "inputMapping",



		class HasTFGraph(Params):
		tfGraph = Param(Params._dummy(), "tfGraph",


		gfn = issn.asGraphFunction(feeds, fetches, strip_and_freeze=True)

		analyzed_df = tfs.analyze(df)

[API] TensorFlow Graph Transformer #39

[API] TensorFlow Graph Transformer #39

Conversation

phi-dbq commented Aug 8, 2017 • edited

What changes are proposed in this pull request?

How is this patch tested?

phi-dbq Aug 8, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Aug 9, 2017 • edited

Codecov Report

sueann left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

phi-dbq commented Aug 9, 2017

Choose a reason for hiding this comment

sueann left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

phi-dbq Aug 22, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

phi-dbq Aug 12, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

phi-dbq Aug 18, 2017 • edited

Choose a reason for hiding this comment

thunterdb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thunterdb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sueann commented Sep 13, 2017

phi-dbq commented Oct 17, 2017

phi-dbq commented Aug 8, 2017 •

edited

phi-dbq Aug 8, 2017 •

edited

codecov-io commented Aug 9, 2017 •

edited

phi-dbq Aug 22, 2017 •

edited

phi-dbq Aug 12, 2017 •

edited

phi-dbq Aug 18, 2017 •

edited