new SchemaDef plugin type to define site-specific schemas (#342)

* new SchemaDef plugin type to define site-specific schemas
* add missing _update_plugin_source for pkg_resources; force schemadefs to load
* add SchemaDef plugin test
* add method to extend all three manifest plugin lists at once
* cleaner save/restore of existing _MANIFEST
* schemadef plugin classes added to otio.schemadef namespace
* Explicitly include our source files in MANIFEST.in

Travis-CI is having some problems reconciling the file list between
sdist and the repo and suggested adding this line:
recursive-include opentimelineio *
so I'm adding it to see if that works.
The particular problem is that it is not picking up one of my latest
additions, namely the schemadef directory and the __init__.py file
in it.  That's the only source file in that directory -- maybe that's
the problem?

* add UnknownSchema object to pass through undefined schemas

Old behavior was to through an exception if any object being read
(or created by instance_from_schema) had an unregistered schema name.
New behavior is to replace the schema of such objects with the
UnknownSchema schema type and save the original schema label in a
field called "UnknownSchemaOriginalLabel".  Such objects are then
available within the OTIO data structure.  The new property
"is_unknown_schema" will be True for these objects and False for all
other SerializableObjects.  When an UnknownSchema is serialized,
the json_serializer will replace the UnknownSchema label with the
original label, so that (for instance) otiocat will pass such
unregistered objects through unchanged.  This is part of the schemadef
project because schemadef plugins imply that we will start to see
site or studio-specific OTIO schema types, and it would be more polite
to pass these through quietly if you don't understand them.

* improve docs and tests for schemadef and unknown schema

MANIFEST.in

@@ -1,5 +1,6 @@
 include README.md CHANGELOG.md LICENSE.txt NOTICE.txt
 recursive-include examples *
+recursive-include opentimelineio *
 
 recursive-exclude docs *
 prune docs
@@ -14,4 +15,4 @@ exclude opentimelineio_contrib/adapters/Makefile
 exclude Makefile
 
 recursive-exclude opentimelineio_contrib/adapters/tests *
-recursive-exclude tests *
+recursive-exclude tests *

docs/index.rst

@@ -47,6 +47,7 @@ Tutorials
    tutorials/wrapping-otio
    tutorials/write-an-adapter
    tutorials/write-a-media-linker
+   tutorials/write-a-schemadef
 
 Use Cases
 ------------
@@ -71,4 +72,3 @@ Indices and tables
 * :ref:`genindex`
 * :ref:`modindex`
 * :ref:`search`
-

docs/tutorials/write-a-schemadef.md

@@ -0,0 +1,118 @@
+# Writing an OTIO SchemaDef Plugin
+
+OpenTimelineIO SchemaDef plugins are plugins that define new schemas within the
+otio type registry system.
+You might want to do this to add new schemas that are specific to your own
+internal studio workflow and shouldn't be part of the generic OpenTimelineIO
+package.
+
+To write a new SchemaDef plugin, you create a Python source file that
+defines and registers one or more new classes subclassed from
+``otio.core.SerializeableObject``.  Multiple schema classes can be defined
+and registered in one plugin, or you can use a separate plugin for each of them.
+
+Here's an example of defining a very simple class called ``MyThing``:
+
+```
+import opentimelineio as otio
+
+@otio.core.register_type
+class MyThing(otio.core.SerializableObject):
+    """A schema for my thing."""
+
+    _serializable_label = "MyThing.1"
+    _name = "MyThing"
+
+    def __init__(
+        self,
+        arg1=None,
+        argN=None
+    ):
+        otio.core.SerializableObject.__init__(self)
+        self.arg1 = arg1
+        self.argN = argN
+
+    arg1 = otio.core.serializable_field(
+        "arg1",
+        doc = ( "arg1's doc string")
+    )
+
+    argN = otio.core.serializable_field(
+        "argN",
+        doc = ( "argN's doc string")
+    )
+
+    def __str__(self):
+        return "MyThing({}, {})".format(
+            repr(self.arg1),
+            repr(self.argN)
+        )
+
+    def __repr__(self):
+        return "otio.schema.MyThing(arg1={}, argN={})".format(
+            repr(self.arg1),
+            repr(self.argN)
+        )
+```
+
+In the example, the ``MyThing`` class has two parameters ``arg1`` and ``argN``,
+but your schema class could have any number of parameters as needed to
+contain the data fields you want to have in your class.
+
+One or more class definitions like this one can be included in a plugin
+source file, which must then be added to the plugin manifest as shown below:
+
+
+## Registering Your SchemaDef Plugin
+
+To create a new SchemaDef plugin, you need to create a Python source file
+as shown in the example above.  Let's call it ``mything.py``.
+Then you must add it to a plugin manifest:
+
+```
+{
+    "OTIO_SCHEMA" : "PluginManifest.1",
+    "schemadefs" : [
+        {
+            "OTIO_SCHEMA" : "MyThing.1",
+            "name" : "mything",
+            "execution_scope" : "in process",
+            "filepath" : "mything.py"
+         }
+    ]
+}
+```
+
+The same plugin manifest may also include adapters and media linkers, if desired.
+
+Then you need to add this manifest to your `$OTIO_PLUGIN_MANIFEST_PATH` environment variable (which is "`:`" separated).
+
+## Using the New Schema in Your Code
+
+Now that we've defined a new otio schema, how can we create an instance of the
+schema class in our code (for instance, in an adapter or media linker)?
+SchemaDef plugins are magically loaded into a namespace called ``otio.schemadef``,
+so you can create a class instance just like this:
+
+```
+import opentimelineio as otio
+
+mine = otio.schemadef.my_thing.MyThing(arg1, argN)
+```
+
+An alternative approach is to use the ``instance_from_schema``
+mechanism, which requires that you create and provide a dict of the parameters:
+
+```
+    mything = otio.core.instance_from_schema("MyThing", 1, {
+        "arg1": arg1,
+        "argN": argN
+    })
+```
+
+This ``instance_from_schema`` approach has the added benefit of calling the
+schema upgrade function to upgrade the parameters in the case where the requested
+schema version is earlier than the current version defined by the schemadef plugin.
+This seems rather unlikely to occur in practice if you keep your code up-to-date,
+so the first technique of creating the class instance directly from
+``otio.schemadef`` is usually preferred.

opentimelineio/__init__.py

@@ -37,6 +37,7 @@
     exceptions,
     core,
     schema,
+    schemadef,
     plugins,
     adapters,
     algorithms,

opentimelineio/core/__init__.py

@@ -27,7 +27,7 @@
 # flake8: noqa
 
 from . import (
-    serializable_object,
+    serializable_object
 )
 from .serializable_object import (
     SerializableObject,
@@ -61,3 +61,7 @@
 from .media_reference import (
     MediaReference,
 )
+from . import unknown_schema
+from .unknown_schema import (
+    UnknownSchema
+)

opentimelineio/core/json_serializer.py

@@ -34,6 +34,8 @@
     type_registry,
 )
 
+from .unknown_schema import UnknownSchema
+
 from .. import (
     exceptions,
     opentime,
@@ -94,6 +96,20 @@ def _encoded_serializable_object(input_otio):
     return result
 
 
+def _encoded_unknown_schema_object(input_otio):
+    orig_label = input_otio.data.get(UnknownSchema._original_label)
+    if not orig_label:
+        raise exceptions.InvalidSerializableLabelError(
+            orig_label
+        )
+    # result is just a dict, not a SerializableObject
+    result = {}
+    result.update(input_otio.data)
+    result["OTIO_SCHEMA"] = orig_label  # override the UnknownSchema label
+    del result[UnknownSchema._original_label]
+    return result
+
+
 def _encoded_time(input_otio):
     return {
         "OTIO_SCHEMA": "RationalTime.1",
@@ -125,6 +141,7 @@ def _encoded_transform(input_otio):
     opentime.RationalTime: _encoded_time,
     opentime.TimeRange: _encoded_time_range,
     opentime.TimeTransform: _encoded_transform,
+    UnknownSchema: _encoded_unknown_schema_object,
     SerializableObject: _encoded_serializable_object,
 }
 

opentimelineio/core/serializable_object.py

@@ -140,6 +140,12 @@ def schema_version(cls):
             cls._serializable_label
         )
 
+    @property
+    def is_unknown_schema(self):
+        # in general, SerializableObject will have a known schema
+        # but UnknownSchema subclass will redefine this property to be True
+        return False
+
     def __copy__(self):
         result = self.__class__()
         result.data = copy.copy(self.data)

opentimelineio/core/type_registry.py

@@ -48,6 +48,12 @@ def schema_version_from_label(label):
     return int(label.split(".")[1])
 
 
+def schema_label_from_name_version(schema_name, schema_version):
+    """Return the serializeable object schema label given the name and version."""
+
+    return "{}.{}".format(schema_name, schema_version)
+
+
 def register_type(classobj, schemaname=None):
     """ Register a class to a Schema Label.
 
@@ -106,9 +112,14 @@ def instance_from_schema(schema_name, schema_version, data_dict):
     """Return an instance, of the schema from data in the data_dict."""
 
     if schema_name not in _OTIO_TYPES:
-        raise exceptions.NotSupportedError(
-            "OTIO_SCHEMA: '{}' not in type registry.".format(schema_name)
-        )
+        from .unknown_schema import UnknownSchema
+
+        # create an object of UnknownSchema type to represent the data
+        schema_label = schema_label_from_name_version(schema_name, schema_version)
+        data_dict[UnknownSchema._original_label] = schema_label
+        unknown_label = UnknownSchema._serializable_label
+        schema_name = schema_name_from_label(unknown_label)
+        schema_version = schema_version_from_label(unknown_label)
 
     cls = _OTIO_TYPES[schema_name]
 

opentimelineio/core/unknown_schema.py

@@ -0,0 +1,43 @@
+#
+# Copyright 2017 Pixar Animation Studios
+#
+# Licensed under the Apache License, Version 2.0 (the "Apache License")
+# with the following modification; you may not use this file except in
+# compliance with the Apache License and the following modification to it:
+# Section 6. Trademarks. is deleted and replaced with:
+#
+# 6. Trademarks. This License does not grant permission to use the trade
+#    names, trademarks, service marks, or product names of the Licensor
+#    and its affiliates, except as required to comply with Section 4(c) of
+#    the License and to reproduce the content of the NOTICE file.
+#
+# You may obtain a copy of the Apache License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the Apache License with the above modification is
+# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the Apache License for the specific
+# language governing permissions and limitations under the Apache License.
+#
+
+"""
+Implementation of the UnknownSchema schema.
+"""
+
+from .serializable_object import SerializableObject
+from .type_registry import register_type
+
+
+@register_type
+class UnknownSchema(SerializableObject):
+    """Represents an object whose schema is unknown to us."""
+
+    _serializable_label = "UnknownSchema.1"
+    _name = "UnknownSchema"
+    _original_label = "UnknownSchemaOriginalLabel"
+
+    @property
+    def is_unknown_schema(self):
+        return True

opentimelineio/plugins/manifest.py

@@ -84,6 +84,7 @@ class Manifest(core.SerializableObject):
     def __init__(self):
         core.SerializableObject.__init__(self)
         self.adapters = []
+        self.schemadefs = []
         self.media_linkers = []
         self.source_files = []
 
@@ -92,16 +93,31 @@ def __init__(self):
         type([]),
         "Adapters this manifest describes."
     )
+    schemadefs = core.serializable_field(
+        "schemadefs",
+        type([]),
+        "Schemadefs this manifest describes."
+    )
     media_linkers = core.serializable_field(
         "media_linkers",
         type([]),
         "Media Linkers this manifest describes."
     )
 
+    def extend(self, another_manifest):
+        """
+        Extend the adapters, schemadefs, and media_linkers lists of this manifest
+        by appending the contents of the corresponding lists of another_manifest.
+        """
+        if another_manifest:
+            self.adapters.extend(another_manifest.adapters)
+            self.schemadefs.extend(another_manifest.schemadefs)
+            self.media_linkers.extend(another_manifest.media_linkers)
+
     def _update_plugin_source(self, path):
         """Track the source .json for a given adapter."""
 
-        for thing in (self.adapters + self.media_linkers):
+        for thing in (self.adapters + self.schemadefs + self.media_linkers):
             thing._json_path = path
 
     def from_filepath(self, suffix):
@@ -140,6 +156,12 @@ def adapter_module_from_name(self, name):
         adp = self.from_name(name)
         return adp.module()
 
+    def schemadef_module_from_name(self, name):
+        """Return the schemadef module associated with a given schemadef name."""
+
+        adp = self.from_name(name, kind_list="schemadefs")
+        return adp.module()
+
 
 _MANIFEST = None
 
@@ -165,8 +187,7 @@ def load_manifest():
                 "contrib_adapters.plugin_manifest.json"
             )
         )
-        result.adapters.extend(contrib_manifest.adapters)
-        result.media_linkers.extend(contrib_manifest.media_linkers)
+        result.extend(contrib_manifest)
     except ImportError:
         pass
 
@@ -194,15 +215,19 @@ def load_manifest():
                         manifest_stream.read().decode('utf-8')
                     )
                     manifest_stream.close()
+                    filepath = pkg_resources.resource_filename(
+                        plugin.module_name,
+                        'plugin_manifest.json'
+                    )
+                    plugin_manifest._update_plugin_source(filepath)
 
             except Exception:
                 logging.exception(
                     "could not load plugin: {}".format(plugin_name)
                 )
                 continue
 
-            result.adapters.extend(plugin_manifest.adapters)
-            result.media_linkers.extend(plugin_manifest.media_linkers)
+            result.extend(plugin_manifest)
     else:
         # XXX: Should we print some kind of warning that pkg_resources isn't
         #        available?
@@ -221,9 +246,11 @@ def load_manifest():
                 continue
 
             LOCAL_MANIFEST = manifest_from_file(json_path)
-            result.adapters.extend(LOCAL_MANIFEST.adapters)
-            result.media_linkers.extend(LOCAL_MANIFEST.media_linkers)
+            result.extend(LOCAL_MANIFEST)
 
+    # force the schemadefs to load and add to schemadef module namespace
+    for s in result.schemadefs:
+        s.module()
     return result