Prototype the custom-scalars plugin. #664

chihuahua · 2017-10-20T05:44:30Z

This plugin makes custom line charts based on regex filtering of tags. Input appreciated! I want to port a prototype into google ASAP so we can test it out.

At a glance, here's how it works. The user sets the dashboard layout via a proto. For instance,

from tensorboard import summary
from tensorboard.plugins.custom_scalar import layout_pb2
...
layout = layout_pb2.Layout(
    category=[
        layout_pb2.Category(
            title='mean biases',
            chart=[
                layout_pb2.Chart(
                    title='mean layer biases',
                    tag=[r'mean/layer\d+/biases'])
            ]),
        layout_pb2.Category(
            title='std weights',
            chart=[
                layout_pb2.Chart(
                    title='stddev layer weights',
                    tag=[r'stddev/layer\d+/weights'])
                ]),
        layout_pb2.Category(
            title='cross entropy ... and maybe some other values',
            chart=[
                layout_pb2.Chart(
                    title='cross entropy',
                    tag=[r'cross entropy']),
                layout_pb2.Chart(
                    title='accuracy',
                    tag=[r'accuracy']),
                layout_pb2.Chart(
                    title='max layer weights',
                    tag=[r'max/layer1/.*', r'max/layer2/.*'])
            ],
            closed=True)
    ])

# Write the Layout proto to disk. This only has to be done once, so we do this outside
# of the TensorFlow graph. `custom_scalars_pb` returns a Summary proto, not an op.
writer.add_summary(summary.custom_scalars_pb(layout))

The set method writes a pbtxt into $LOG_DIR/custom_scalars_config.pbtxt, which the plugin uses to organize the dashboard:

Different tags within the same runs are differentiated by markers. The user can opt for certain categories to be opened by default.

Again, I really want to know everyone's feedback on this. This prototype deviates a bit from the discussion on the thread, but I think makes sense and addresses @georgedahl's needs.

I was wavering between whether to pass the dashboard organization via a flag or via a programmatic API ... and opted for the latter because

The user can more straightforwardly specify the dashboard layout via creating a proto.
A programmatic API lets a script change the layout over time.
Adding a flag for custom line charts introduces some dilemmas. For instance, what if both a flag value is specified and the pbtxt is written to disk? Which one should be the source of truth?

jart · 2017-10-20T20:13:35Z

I like this so far. Here's some feedback.

Maybe have this:

category {
  title: "mean biases"
  regex: "^mean/layer\\d+/biases$"
}

Be this instead:

category {
  title: "mean biases"
  chart: {
    name: "Mean Layer Biases"
    tag: "mean/layer\\d+/biases"
  }
}

Name is optional and defaults to tag_regex
Tag is repeated
Match whole strings by default, so no ugly ^$ in average case
We can add new ways to filter in the future, possibly even based on chart data.

On the UI front, I like the idea of not having the search box and match list be there by default. The matching tag names can be viewed in the hover box. But I like the customizability. Maybe a click dialog or a sliding drawer?

jart · 2017-10-20T20:14:33Z

Also super helpful instructions in the plugin inactive page, that show example proto files (in addition to Python API examples) would really help users bootstrap themselves with this feature. In that sense, I think you were correct that there are advantages to this being its own plugin.

teamdandelion

I'd like to see the config placed in a way that we could imagine generalizing to many plugins. If we had 10 plugins, all dumping their random config files into the logdir, it will be a mess. So I think we should standardize on something like:

dir/plugins/custom_scalars/foo.pbtxt

Also, I think the plugin asset should be handled by framework code that interfaces thru the SummaryWriter, rather than directly writing files. This will improve consistency across plugins, and interoperability for when we are writing data that isn't going to a local directory on disk, but to a distributed backend / tensorbase / what have you.

Plugin Asset Util may be relevant.

Also: What should the behavior be if we have

RunA defines LayoutA in ./runs/A
RunB defines LayoutB in ./runs/B
User decides to look at both runs by pointing to ./runs

teamdandelion · 2017-10-20T22:26:58Z

tensorboard/components/vz_line_chart/vz-line-chart.ts

+    // Downsample the data. Otherwise, too many markers clutter the chart.
+    const skipLength = Math.ceil(originalData.length / _MAX_MARKERS);
+    const data = new Array(Math.floor(originalData.length / skipLength));
+    let indexIntoOriginal = 0;


indexIntoOriginal is unused

teamdandelion · 2017-10-20T22:27:19Z

tensorboard/components/vz_line_chart/vz-line-chart.ts

+    for (let i = 0, j = 0; i < data.length; i++, j += skipLength) {
+      data[i] = originalData[j];
+    }
+    return new Plottable.Dataset(data, {name: original.metadata().name});


Why not copy or reuse the whole metadata object?

Good point. Done.

teamdandelion · 2017-10-20T23:54:48Z

tensorboard/plugins/custom_scalar/custom_scalars_plugin.py

+    """
+    self._logdir = context.logdir
+    self._multiplexer = context.multiplexer
+    self._scalars_plugin = scalars_plugin.ScalarsPlugin(context)


Note: we're counting on instantiating a scalars plugin not to have side effects. i think we can make this assumption, since the plugin will re-use the multiplexer and plugins don't do anything directly if you don't request their WSGI apps.

Hm, but the custom scalars plugin itself does have a side effect (starting a config file checking thread) as soon as it's constructed. Maybe it would be worth formalizing a little when one plugin depends on another, e.g. by refactoring the scalars plugin to have a data-loading class without any of the WSGI stuff that is what other plugins should depend on directly?

Just a note: the distributions plugin similarly instantiates a histograms plugin. At the time, @dandelionmane and I agreed that this was a Good Thing because they depend on the same data source and this code makes the data dependency especially explicit.

Yeah, per @wchargin, we have a precedence, although as @nfelt noted, plugin authors may one day add side effects (like threads) without being aware of their full impact.

I recently submitted #659 - the latest commit to this PR makes use of the mapping.

teamdandelion · 2017-10-20T23:57:05Z

tensorboard/plugins/custom_scalar/custom_scalars_plugin.py

+          layout.CONFIG_FILE, e)
+
+    # Wait a while before checking again.
+    time.sleep(_CONFIG_FILE_CHECK_THROTTLE)


I think this method is misleadingly named. It is called "periodically_check" but it doesn't implement the periodic check - really it is "check then delay".

I think the time.sleep should be factored to live with (or include) the while True: loop for checking the file. The try/except error handling and logging can be moved into check_for_config_file.

teamdandelion · 2017-10-21T00:00:19Z

tensorboard/plugins/custom_scalar/custom_scalars_plugin.py

+
+    layout_proto = scalars_layout_pb2.Layout()
+    text_format.Parse(pbtxt_contents, layout_proto)
+    self._layout = json_format.MessageToJson(


I think while in Python-land it should be stored as a proto, not JSON. Early conversion to JSON just loses inspectability/safety, no?

Then, you can convert to JSON in the handler when it is actually being sent to the frontend.

It gives a straightforward mapping between data storage type and location:

fronted = json
backend = proto

Done. That general rule sounds great, albeit we do incur a little latency from serialization at response time.

teamdandelion · 2017-10-21T00:07:02Z

tensorboard/plugins/custom_scalar/layout.py

+
+CONFIG_FILE = 'custom_scalars_config.pbtxt'
+
+def set_layout(logdir, scalars_layout):


Better to pass some SummaryWriter-esque object rather than a logdir, since the SummaryWriter is the canonical point by which the summary data is exported.

Reasons:

One often keeps access to a SummaryWriter when writing code that deals with summaries, but the logdir might be in some outer scope

In the future, we may switch to e.g a DatabseWriter or RemoteServerWriter and in these cases there isn't meaningfully a "logdir". It would be better to implement a consistent interface across all our writers that can handle this, rather than privilging the logdir concept.

I see what you mean ... I'm grappling with how file writer pertains to a run. A config for a run is meaningless: We only have 1 config for the dashboard. Maybe we accept a logdir for now (and pass it into plugin_asset_util.PluginDirectory()? And then we do away with the concept of logdir when say we save the config in SQL?

teamdandelion · 2017-10-21T00:07:39Z

tensorboard/plugins/custom_scalar/layout_test.py

+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""Integration tests for the pr_curves plugin."""


s/pr_curves/custom_scalars/

nfelt · 2017-10-21T01:55:18Z

tensorboard/plugins/custom_scalar/custom_scalars_plugin.py

+    """
+    self._logdir = context.logdir
+    self._multiplexer = context.multiplexer
+    self._scalars_plugin = scalars_plugin.ScalarsPlugin(context)


Hm, but the custom scalars plugin itself does have a side effect (starting a config file checking thread) as soon as it's constructed. Maybe it would be worth formalizing a little when one plugin depends on another, e.g. by refactoring the scalars plugin to have a data-loading class without any of the WSGI stuff that is what other plugins should depend on directly?

nfelt · 2017-10-21T01:58:14Z

tensorboard/plugins/custom_scalar/custom_scalars_plugin.py

+
+    # Start a separate thread that periodically checks for the config file
+    # specifying layout.
+    threading.Thread(target=self._periodically_check_for_config_file).start()


Can you give the thread a name? Makes it easier to follow in the python logs when debugging. E.g. "CustomScalarsConfigThread" or something.

Done. That does make logs for the thread easier to follow!

natashamjaques · 2017-10-23T19:32:44Z

This looks like a great new feature. I just wanted to echo Dandelion's question about whether/how this feature would make it possible to compare line charts from two different models (i.e. RunA defines LayoutA in ./runs/A, RunB defines LayoutB in ./runs/B). I'm interested in comparing multiple models that may have slightly different graphs (e.g. model A is the baseline, model B has some added layer or new component).

Beyond that, the models might actually have several runs each (I'm thinking several workers in A3C), so ideally I'd love to get a mean and confidence interval around all the runs for model A and plot that against the mean and CI for model B. I definitely agree with @georgedahl's comment in #597; I'm also currently pulling out all the data from the tf.events and plotting it myself in a notebook, in order to generate something like this:

wchargin · 2017-10-24T01:59:53Z

Happy to see that you're working on this!

Just wanted to let you know that I’ve seen this and have read (and will continue to follow) the discussion, but I don’t have the bandwidth to do a 1640-line code review right now. If there are future, smaller PRs that you’d like me to look at—or anything specific in this PR—please feel free to re-add me.

I generally agree with comments so far (especially defaulting to the ^$ anchors and keeping things as structured data as long as possible within Python—aim for strongly typed, not stringly typed :-) ).

chihuahua

@jart, wonderful ideas! I incorporated the changes. Let me know what you think.

The one change I did not incorporate yet is a helpful "no data" message. I plan to add that once we settle on the API for writing/reading the config file.

I made each Category contain several Charts. Each Chart contains several tags (or really regexes for tags).
We match whole strings by default.
The list of matches is now collapsible.

@dandelionmane, I addressed some of your detailed comments, but I want to discuss the system for loading plugin-specific files.

I think we can use the PluginDirectory method here:
https://github.com/tensorflow/tensorboard/blob/master/tensorboard/backend/event_processing/plugin_asset_util.py
Specifically, we can have plugins call that method and then be responsible for writing data into it.

This would involve preserving the concept of logdir though, which I think might be necessary for now: The config file is only relevant for the top-level log directory (which concerns your other question), while each file writer pertains to a run (I think per-run config files should be ignored ... what would a per-run config file mean?).

@natashamjaques, after this plugin goes in, each run for each model should manifest as a distinct run in TensorBoard if you start TensorBoard at the top-level logdir. TensorBoard recursively finds runs within the logdir - you'll be able to explicitly select those runs and they should show within the same custom scalars plot.

Regarding confidence intervals, we would actually need to build a separate plugin since the scalars summary format is rigid. I am currently adding frontend logic to support visualizing confidence intervals though. See this separate effort:
#608 (comment)

chihuahua · 2017-10-24T02:56:49Z

tensorboard/components/vz_line_chart/vz-line-chart.ts

+    // Downsample the data. Otherwise, too many markers clutter the chart.
+    const skipLength = Math.ceil(originalData.length / _MAX_MARKERS);
+    const data = new Array(Math.floor(originalData.length / skipLength));
+    let indexIntoOriginal = 0;


chihuahua · 2017-10-24T02:57:18Z

tensorboard/components/vz_line_chart/vz-line-chart.ts

+    for (let i = 0, j = 0; i < data.length; i++, j += skipLength) {
+      data[i] = originalData[j];
+    }
+    return new Plottable.Dataset(data, {name: original.metadata().name});


Good point. Done.

chihuahua · 2017-10-24T02:59:07Z

tensorboard/plugins/custom_scalar/custom_scalars_plugin.py

+
+    # Start a separate thread that periodically checks for the config file
+    # specifying layout.
+    threading.Thread(target=self._periodically_check_for_config_file).start()


Done. That does make logs for the thread easier to follow!

chihuahua · 2017-10-24T03:02:39Z

tensorboard/plugins/custom_scalar/custom_scalars_plugin.py

+
+    layout_proto = scalars_layout_pb2.Layout()
+    text_format.Parse(pbtxt_contents, layout_proto)
+    self._layout = json_format.MessageToJson(


Done. That general rule sounds great, albeit we do incur a little latency from serialization at response time.

chihuahua · 2017-10-24T03:11:59Z

tensorboard/plugins/custom_scalar/layout.py

+
+CONFIG_FILE = 'custom_scalars_config.pbtxt'
+
+def set_layout(logdir, scalars_layout):


I see what you mean ... I'm grappling with how file writer pertains to a run. A config for a run is meaningless: We only have 1 config for the dashboard. Maybe we accept a logdir for now (and pass it into plugin_asset_util.PluginDirectory()? And then we do away with the concept of logdir when say we save the config in SQL?

chihuahua · 2017-10-24T03:12:22Z

tensorboard/plugins/custom_scalar/layout_test.py

+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""Integration tests for the pr_curves plugin."""


chihuahua · 2017-10-24T03:40:33Z

tensorboard/plugins/custom_scalar/custom_scalars_plugin.py

+    """
+    self._logdir = context.logdir
+    self._multiplexer = context.multiplexer
+    self._scalars_plugin = scalars_plugin.ScalarsPlugin(context)


Yeah, per @wchargin, we have a precedence, although as @nfelt noted, plugin authors may one day add side effects (like threads) without being aware of their full impact.

I recently submitted #659 - the latest commit to this PR makes use of the mapping.

wchargin · 2017-10-25T12:07:00Z

tensorboard/plugins/custom_scalar/custom_scalars_plugin.py

+    }
+
+    The response is an empty object if no layout could be found.
+    """  # pylint: disable=anomalous-backslash-in-string


…why? What's wrong with using a raw docstring (r"""Fetches…""") or properly escaping the metacharacter (\\d+/biases)?

Using a raw doc string sounds great! Done.

Some plugins (such as the `custom-scalar` and `debugger` plugins) make use of the `vz_line_chart` component but plot data series that are not runs (such as custom run-tag combos or tensor values). This change makes the `tf-line-chart-data-loader` component accept a data series property that defaults to the list of runs. Test plan: Note that the custom scalars plugin works (allows for custom run-tag combos within `vz_line_chart`) after this change: #664

chihuahua · 2017-11-06T20:22:18Z

A quick update: @dandelionmane and I discussed. I plan to make set_layout take a SummaryWriter and write the Layout proto into a summary. As @jart had also noted, this makes use of the plugin system and ports well with TensorBase. We may migrate towards using the new concept of an experiment later.

We had discussed simplifying the Layout proto and avoiding categorization for this dashboard to simplify the API ... but later I find @jart's point compelling: If we document this plugin well, the user should just be able to copy and paste a sample configuration for usage. Hence, I plan to keep categorization for now to give users that flexibility.

jart · 2017-11-07T01:35:20Z

SGTM

chihuahua · 2017-11-10T19:59:32Z

FYI, I am stuck by googlebot:

Usually, rebasing does away with this issue, but not this time. Will keep trying things.

jart

I think I'm falling in love with this change. The amount of thought and effort put into this really shows. I honestly can't find much wrong with it. We should be able to get this submitted soon.

jart · 2017-11-11T02:59:24Z

tensorboard/plugins/custom_scalar/summary.py

+    scalars_layout: The scalars_layout_pb2.Layout proto that specifies the
+        layout.
+  """
+  tensor = tf.make_tensor_proto(


Is it possible to have a precondition here to check if it's the right type of proto?

jart · 2017-11-11T03:00:04Z

tensorboard/plugins/custom_scalar/summary.py

+  Args:
+    scalars_layout: The scalars_layout_pb2.Layout proto that specifies the
+        layout.
+  """


Please document that it Returns: a tf.Summary proto that can be passed to FileWriter.

jart · 2017-11-11T03:01:16Z

tensorboard/plugins/custom_scalar/tf_custom_scalar_dashboard/tf-custom-scalar-dashboard.html

+from tensorboard.plugins.custom_scalar import layout_pb2
+...
+with tf.summary.FileWriter(LOGDIR) as file_writer:
+  file_writer.add_summary(


I love this doc but it needs to be updated due to recent changes.

Done. We now use summary.custom_scalars_pb.

Yes but FileWriter isn't going to be used in go/tf-summaries-2.0. I guess we can change this doc later.

OK, lets just leave out the part about constructing a FileWriter. Done.

jart · 2017-11-11T03:06:25Z

tensorboard/plugins/custom_scalar/tf_custom_scalar_dashboard/tf-custom-scalar-card.html

+              as="seriesName"
+            >
+              <div
+                  class="match-list-entry"


go/htmlstyle should, generally speaking, take precedence when creating new files. There's no stylistic disadvantage to putting this on the previous line and then aligning the attrs.

jart · 2017-11-11T03:08:01Z

tensorboard/plugins/custom_scalar/tf_custom_scalar_dashboard/tf-custom-scalar-card.html

+              is="dom-repeat"
+              items="[[_dataSeriesStrings]]"
+              as="seriesName"
+            >


I'd say put this on the previous line, since nothing comes after the >.

chihuahua

Thank you Justine for the input and good ideas - I'm also excited to bring this to users!

chihuahua · 2017-11-13T21:36:00Z

tensorboard/plugins/custom_scalar/summary.py

+  Args:
+    scalars_layout: The scalars_layout_pb2.Layout proto that specifies the
+        layout.
+  """


chihuahua · 2017-11-13T21:38:05Z

tensorboard/plugins/custom_scalar/summary.py

+    scalars_layout: The scalars_layout_pb2.Layout proto that specifies the
+        layout.
+  """
+  tensor = tf.make_tensor_proto(


chihuahua · 2017-11-13T21:46:14Z

tensorboard/plugins/custom_scalar/tf_custom_scalar_dashboard/tf-custom-scalar-dashboard.html

+from tensorboard.plugins.custom_scalar import layout_pb2
+...
+with tf.summary.FileWriter(LOGDIR) as file_writer:
+  file_writer.add_summary(


Done. We now use summary.custom_scalars_pb.

chihuahua · 2017-11-13T21:48:06Z

tensorboard/plugins/custom_scalar/tf_custom_scalar_dashboard/tf-custom-scalar-card.html

+              is="dom-repeat"
+              items="[[_dataSeriesStrings]]"
+              as="seriesName"
+            >


chihuahua · 2017-11-13T21:48:09Z

tensorboard/plugins/custom_scalar/tf_custom_scalar_dashboard/tf-custom-scalar-card.html

+              as="seriesName"
+            >
+              <div
+                  class="match-list-entry"


chihuahua · 2017-11-13T21:53:23Z

Also, if we could make cla/google go green, that'd be great.

jart · 2017-11-14T16:37:38Z

tensorboard/plugins/custom_scalar/tf_custom_scalar_dashboard/tf-custom-scalar-dashboard.html

+from tensorboard.plugins.custom_scalar import layout_pb2
+...
+with tf.summary.FileWriter(LOGDIR) as file_writer:
+  file_writer.add_summary(


Yes but FileWriter isn't going to be used in go/tf-summaries-2.0. I guess we can change this doc later.

This plugin makes custom line charts based on regex filtering of tags.

Sorry, I had inadvertently introduced an empty foo file into the TensorBoard directory with PR tensorflow#664.

Sorry, I had inadvertently introduced an empty foo file into the TensorBoard directory with PR #664.

It has always had wrong property value since tensorflow#664.

Cause: ignoreYOutlier instead of _ignoreYOutlier It had wrong property value since #664.

googlebot · 2021-12-26T23:02:25Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

chihuahua added the type:feature (new plugin) label Oct 20, 2017

chihuahua requested review from jart, teamdandelion and nfelt October 20, 2017 05:44

chihuahua force-pushed the custom-scalars branch 2 times, most recently from a0b1b99 to dca1a06 Compare October 20, 2017 06:10

chihuahua requested a review from wchargin October 20, 2017 18:48

chihuahua mentioned this pull request Oct 20, 2017

Allow grouping of multiple runs #376

Open

teamdandelion reviewed Oct 21, 2017

View reviewed changes

nfelt reviewed Oct 21, 2017

View reviewed changes

wchargin removed their request for review October 24, 2017 02:00

chihuahua commented Oct 24, 2017

View reviewed changes

wchargin reviewed Oct 25, 2017

View reviewed changes

chihuahua mentioned this pull request Oct 26, 2017

Allow for data series that are not runs #679

Merged

chihuahua force-pushed the custom-scalars branch from 0f00c29 to e2f2013 Compare November 1, 2017 00:41

chihuahua force-pushed the custom-scalars branch 5 times, most recently from b83cc81 to 8ab307b Compare November 6, 2017 20:15

chihuahua force-pushed the custom-scalars branch from 8ab307b to 6d24ec7 Compare November 6, 2017 20:37

chihuahua force-pushed the custom-scalars branch 2 times, most recently from 1cfda30 to d2a766e Compare November 7, 2017 02:45

chihuahua force-pushed the custom-scalars branch from f712947 to 7f21a39 Compare November 10, 2017 09:20

chihuahua force-pushed the custom-scalars branch 3 times, most recently from 0a83d62 to e523c78 Compare November 11, 2017 00:30

chihuahua force-pushed the custom-scalars branch from e523c78 to 49cfe40 Compare November 11, 2017 00:33

jart suggested changes Nov 11, 2017

View reviewed changes

chihuahua force-pushed the custom-scalars branch from 49cfe40 to 2ccf6d5 Compare November 13, 2017 21:51

chihuahua commented Nov 13, 2017

View reviewed changes

jart approved these changes Nov 14, 2017

View reviewed changes

chihuahua force-pushed the custom-scalars branch from 2ccf6d5 to f16d309 Compare November 15, 2017 00:06

Prototype the custom-scalars plugin.

10a76d6

This plugin makes custom line charts based on regex filtering of tags.

chihuahua force-pushed the custom-scalars branch from f16d309 to 10a76d6 Compare November 15, 2017 01:38

chihuahua merged commit f755228 into tensorflow:master Nov 15, 2017

chihuahua deleted the custom-scalars branch November 15, 2017 01:56

chihuahua added a commit to chihuahua/tensorboard that referenced this pull request Nov 15, 2017

Remove empty foo file

9c81e07

Sorry, I had inadvertently introduced an empty foo file into the TensorBoard directory with PR tensorflow#664.

chihuahua mentioned this pull request Nov 15, 2017

Remove empty foo file #737

Merged

chihuahua added a commit that referenced this pull request Nov 15, 2017

Remove empty foo file (#737)

9833331

Sorry, I had inadvertently introduced an empty foo file into the TensorBoard directory with PR #664.

nfelt added the plugin:custom-scalars label Jan 3, 2018

nfelt mentioned this pull request Jan 3, 2018

Feature request: plot different scalar summaries together based on regexes #300

Closed

CatherineSue mentioned this pull request May 26, 2018

Add save graph and tensor logging to tensorboard ryanjulian/rllab#88

Merged

Joshua1989 mentioned this pull request Jun 8, 2018

Is it possible to use Prototype the custom-scalars plugin in tensorflow? lanpa/tensorboardX#161

Closed

stephanwlee added a commit to stephanwlee/tensorboard that referenced this pull request Oct 10, 2018

Connect ignoreYOutlier in custom scalars

ceb0926

It has always had wrong property value since tensorflow#664.

stephanwlee mentioned this pull request Oct 10, 2018

Connect ignoreYOutlier in custom scalars #1508

Merged

stephanwlee added a commit that referenced this pull request Oct 10, 2018

Connect ignoreYOutlier in custom scalars (#1508)

8c90121

Cause: ignoreYOutlier instead of _ignoreYOutlier It had wrong property value since #664.

googlebot added the cla: yes label Dec 26, 2021


		CONFIG_FILE = 'custom_scalars_config.pbtxt'

		def set_layout(logdir, scalars_layout):

Prototype the custom-scalars plugin. #664

Prototype the custom-scalars plugin. #664

Conversation

chihuahua commented Oct 20, 2017 • edited

jart commented Oct 20, 2017

jart commented Oct 20, 2017

teamdandelion left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

natashamjaques commented Oct 23, 2017

wchargin commented Oct 24, 2017

chihuahua left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chihuahua commented Nov 6, 2017

jart commented Nov 7, 2017

chihuahua commented Nov 10, 2017

jart left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chihuahua left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chihuahua commented Nov 13, 2017

Choose a reason for hiding this comment

googlebot commented Dec 26, 2021

chihuahua commented Oct 20, 2017 •

edited