nPMI Plugin Backend (experimental) (tensorflow#3799)

* Added nPMI Plugin Backend Files

Added files for the nPMI plugin backend. This is not yet completely
working.

* Added Comment to Active Function

Commented the isActive function of the plugin.

* Fix Plugin Test File

The test file did not work correctly as of some indentation issue. THis
has been fixed now.

* Summary Tests

Added tests for the summary writer.

* Corrected Indentation for Main File

Corrected the indentation for the main plugin backend file.

* Added Missing License

Added a missing license disclaimer to the summary test file.

* More Indentation Correction

Corrected more of the indentation issues in the python files.

* Further Linter Issues

Fixed further linter issues.

* Linting Correction

Further Linting Correction

* Further Linting

More linting changes.

* Fixed BUILD File

Fixed linter error for BUILD file.

* Fixed Test Issue on TF 1.15

Fixed failing tests because of wrong imports on TF 1.15.

* Rename NPMI to Npmi in Classes

For consistency, this renaming has been done.

* Missed Rename

Missed renaming in one place.

* More Explanation for the Plugin

Added more explanation for the plugin with a readme and one more comment.

* Removed Keys and added Comments

Removed some keys that are not really needed for the plugin and added
more comments to clarify routes.

* Build Deps Cleaned

Cleaned up a little in the build deps.

* Removed Unused Imports

Had several imports that were not used. Removed them now.

* Added Dependency

Added a missing dependency for the plugin.

* Changed dependencies to actual Import

Changed dependecies of the summary writer to the actual imports of this
file.

* Restructured Summary Writer

Summary writer now with three different functions for the routes.

* Fixed Linter Issues

After this restructure, I introduced some linter issues, which are now
fixed.

* Added Safe JSON Encoder

Added a encoder that safely handles numpy nan, inf, and -inf values.

* Linting Fixes

After adding the SafeEncoder, needed some linter fixes.

* Final Linter Change

One more Linter change for the safe encoder.

* Added Proto for Metadata

Added aproto for the metadata field.

* Metadata is Own File

Now, the metadata is its own file. This had to be done for the metadata
content is a proto now and thus metadata is accessed at different
places.

* Begin data_provider Transition

Beginning the transition from multiplexer to data_provider.

* Extra Whitespace Bug

Removed an extra whitespace where there should not be one.

* Plugin Name from Metadata

Now getting the plugin name from metadata to avoid duplication.

* Fix BUILD File

Some fixes in the BUILD file for the CL.

* Added Version Field

Added a version field to the metadata.

* Switch from Multiplexer to Data Provider

Switched from multiplexter to using data provider for serving data.

* Linting

Corrected linting issues.

* More Linting

More linting corrections.

* Removed Safe Encoder for Parse Function

Now parsing the values that we get from a tensor and converting nan to
None.

* Fixed Linting in Convert Function

The convert function now is correctly formatted as well.

* Minor Comment Changes

Changed some comments for better explanation.

* Removed Table Field

Removed the table field from all the data.

* Linting Fix

Fixed a minor linter error.

* Moved To Correct Directory

The backend has been moved to the tensorboard/plugins directory.

* Build Lint Fix

* Fixed Wrong Import

The default.py was still using the old path.

Co-authored-by: Alex Bäuerle <abauerle@google.com>

tensorboard/BUILD

@@ -270,6 +270,7 @@ py_library(
         "//tensorboard/plugins/hparams:hparams_plugin",
         "//tensorboard/plugins/image:images_plugin",
         "//tensorboard/plugins/mesh:mesh_plugin",
+        "//tensorboard/plugins/npmi:npmi_plugin",
         "//tensorboard/plugins/pr_curve:pr_curves_plugin",
         "//tensorboard/plugins/profile_redirect:profile_redirect_plugin",
         "//tensorboard/plugins/scalar:scalars_plugin",

tensorboard/default.py

@@ -53,6 +53,7 @@
 from tensorboard.plugins.text import text_plugin
 from tensorboard.plugins.text_v2 import text_v2_plugin
 from tensorboard.plugins.mesh import mesh_plugin
+from tensorboard.plugins.npmi import npmi_plugin
 
 
 logger = logging.getLogger(__name__)
@@ -66,6 +67,14 @@ class ExperimentalTextV2Plugin(
     pass
 
 
+class ExperimentalNpmiPlugin(
+    npmi_plugin.NpmiPlugin, experimental_plugin.ExperimentalPlugin
+):
+    """Angular nPMI Plugin marked as experimental."""
+
+    pass
+
+
 # Ordering matters. The order in which these lines appear determines the
 # ordering of tabs in TensorBoard's GUI.
 _PLUGINS = [
@@ -86,6 +95,7 @@ class ExperimentalTextV2Plugin(
     hparams_plugin.HParamsPlugin,
     mesh_plugin.MeshPlugin,
     ExperimentalTextV2Plugin,
+    ExperimentalNpmiPlugin,
 ]
 
 

tensorboard/plugins/npmi/BUILD

@@ -0,0 +1,90 @@
+# Description:
+# TensorBoard nPMI Plugin: A plugin for inspecting nPMI calculations for trained
+# models. nPMI is a technique presented in a paper (https://svn.spraakdata.gu.se/repos/gerlof/pub/www/Docs/npmi-pfd.pdfhttps://svn.spraakdata.gu.se/repos/gerlof/pub/www/Docs/npmi-pfd.pdf)
+# presented by Gerlof Bouma.
+
+load("//tensorboard/defs:protos.bzl", "tb_proto_library")
+
+package(default_visibility = ["//tensorboard:internal"])
+
+licenses(["notice"])
+
+py_library(
+    name = "npmi_plugin",
+    srcs = ["npmi_plugin.py"],
+    srcs_version = "PY2AND3",
+    deps = [
+        ":metadata",
+        "//tensorboard:errors",
+        "//tensorboard:plugin_util",
+        "//tensorboard/backend:http_util",
+        "//tensorboard/plugins:base_plugin",
+        "//tensorboard/util:tensor_util",
+        "@org_pocoo_werkzeug",
+    ],
+)
+
+py_test(
+    name = "npmi_plugin_test",
+    size = "small",
+    srcs = ["npmi_plugin_test.py"],
+    main = "npmi_plugin_test.py",
+    srcs_version = "PY2AND3",
+    deps = [
+        ":npmi_plugin",
+        ":summary",
+        "//tensorboard:context",
+        "//tensorboard:expect_tensorflow_installed",
+        "//tensorboard:plugin_util",
+        "//tensorboard/backend/event_processing:data_provider",
+        "//tensorboard/backend/event_processing:event_multiplexer",
+        "//tensorboard/plugins:base_plugin",
+        "//tensorboard/util:test_util",
+    ],
+)
+
+py_library(
+    name = "summary",
+    srcs = ["summary.py"],
+    srcs_version = "PY2AND3",
+    visibility = [
+        "//visibility:public",
+    ],
+    deps = [
+        ":metadata",
+        "//tensorboard/compat",
+    ],
+)
+
+py_test(
+    name = "summary_test",
+    size = "small",
+    srcs = ["summary_test.py"],
+    srcs_version = "PY2AND3",
+    deps = [
+        ":metadata",
+        ":summary",
+        "//tensorboard:expect_tensorflow_installed",
+        "//tensorboard/compat",
+        "//tensorboard/util:test_util",
+    ],
+)
+
+py_library(
+    name = "metadata",
+    srcs = ["metadata.py"],
+    srcs_version = "PY2AND3",
+    visibility = [
+        "//tensorboard:internal",
+    ],
+    deps = [
+        ":protos_all_py_pb2",
+        "//tensorboard/compat/proto:protos_all_py_pb2",
+        "//tensorboard/util:tb_logging",
+    ],
+)
+
+tb_proto_library(
+    name = "protos_all",
+    srcs = ["plugin_data.proto"],
+)

tensorboard/plugins/npmi/README.md

@@ -0,0 +1,5 @@
+# nPMI Plugin
+
+The nPMI plugin is a frontend to the techniques presented in a [paper](https://svn.spraakdata.gu.se/repos/gerlof/pub/www/Docs/npmi-pfd.pdf) by Gelof Bouma.
+
+The plugin is currently experimental and under active development.

tensorboard/plugins/npmi/http_api.md

@@ -0,0 +1,92 @@
+# nPMI plugin HTTP API
+
+The npmi plugin name is `npmi`, so all its routes are under `data/plugin/npmi`.
+
+## `data/plugin/npmi/tags`
+
+Retrieves an index of tags that were written to the summary. Tags are from a
+fixed set of known tag names. The table object is used as it is possible for the
+exporter that we use to export multiple tables at the same time. Table names can
+be anything user-defined.
+
+Returns a dictionary mapping from `runName` to an array of dictionaries mapping
+from `tagName` to the table they were exported for. Example:
+
+```json
+{
+  "run_1": {
+    "metric_annotations": {
+      "table": "label_statistics"
+    },
+    "metric_classes": {
+      "table": "label_statistics"
+    },
+    "metric_results": {
+      "table": "label_statistics"
+    }
+  }
+}
+```
+
+## `data/plugin/npmi/annotations`
+
+Retrieves all the annotations that were written to the summary.
+
+Returns a dictionary mapping from `runName` to an array of annotations. Example:
+
+```json
+{
+  "run_1": [
+    "tree",
+    "field",
+    "road",
+  ]
+}
+```
+
+## `data/plugin/npmi/metrics`
+
+Retrieves all the metrics that were calculated.
+
+Returns a dictionary mapping from `runName` to an array of metrics. Example:
+
+```json
+{
+  "run_1": [
+    "npmi@green",
+    "npmi@blue",
+    "npmi@red",
+  ]
+}
+```
+
+## `data/plugin/npmi/values`
+
+Retrieves all the values that were calculated. Values are the actual results of
+the nPMI calculations.
+
+Returns a dictionary mapping from `runName` to an array of arrays of metric
+values. In this, the outer array indices resolve to the annotations, whereas the
+inner array resolves to the metrics per annotation. Example:
+
+```json
+{
+  "run_1": [
+    [
+      0.34362471103668213,
+      0.17068558931350708,
+      0.28587889671325684,
+    ],
+    [
+      0.17293912172317505,
+      0.05774582177400589,
+      -0.17293912172317505,
+    ],
+    [
+      -0.11519329994916916,
+      -0.05774582177400589,
+      0.11519329994916916
+    ]
+  ]
+}
+```

tensorboard/plugins/npmi/metadata.py

@@ -0,0 +1,70 @@
+# -*- coding: utf-8 -*-
+# Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+from tensorboard.compat.proto import summary_pb2
+
+from tensorboard.plugins.npmi import plugin_data_pb2
+from tensorboard.util import tb_logging
+
+logger = tb_logging.get_logger()
+
+PLUGIN_NAME = "npmi"
+
+# The most recent value for the `version` field of the
+# `ScalarPluginData` proto.
+PROTO_VERSION = 0
+
+ANNOTATIONS_TAG = "_npmi_/annotations"
+METRICS_TAG = "_npmi_/metrics"
+VALUES_TAG = "_npmi_/values"
+
+
+def create_summary_metadata(description):
+    content = plugin_data_pb2.NpmiPluginData(version=PROTO_VERSION)
+    return summary_pb2.SummaryMetadata(
+        summary_description=description,
+        plugin_data=summary_pb2.SummaryMetadata.PluginData(
+            plugin_name=PLUGIN_NAME, content=content.SerializeToString(),
+        ),
+        data_class=summary_pb2.DATA_CLASS_TENSOR,
+    )
+
+
+def parse_plugin_metadata(content):
+    """Parse summary metadata to a Python object.
+    Arguments:
+      content: The `content` field of a `SummaryMetadata` proto
+        corresponding to the scalar plugin.
+    Returns:
+      A `ScalarPluginData` protobuf object.
+    """
+    if not isinstance(content, bytes):
+        raise TypeError("Content type must be bytes")
+    result = plugin_data_pb2.NpmiPluginData.FromString(content)
+    if result.version == 0:
+        return result
+    else:
+        logger.warning(
+            "Unknown metadata version: %s. The latest version known to "
+            "this build of TensorBoard is %s; perhaps a newer build is "
+            "available?",
+            result.version,
+            PROTO_VERSION,
+        )
+        return result