Release/2.1.2 (#522)

* Add service_identity module dependency

Removes a test warning, improves twisted's ability to verify certificates.

* Fix #474

Use relative links to templates.
Put message payload into the UIP display so that the receiver can read it without having to open the message XML up.

* Remove requirement for pytest-capturelog, removed from

No longer needed because it has been merged into the core.

* Refactor proxy handling for client

Track a change to proxy handling in Autobahn / twisted for #477. There's probably a more elegant way, but this works for now... Lacking a test, but have verified it locally.

* More elegant fix for proxy

We don't need the HTTPProxyConfig object at all anymore, just a `dict` will do.

* Don't call fixtures directly

Closes #490.

Calling fixtures directly is deprecated, this solution described at
https://docs.pytest.org/en/latest/deprecations.html#calling-fixtures-directly
seems to work, creating a named fixture rather than defining the "then"
step as a fixture directly.

* Restrict autobahn to versions before 18

Temporarily fixes #496, prior to addressing the problem properly anyway.

* Restrict autobahn to versions before 18

Temporarily fixes #496, prior to addressing the problem properly anyway.

* Issue 447 resequencer documentation (#515)

* Fix indentation warning

* Fix underline length warning

* Fix document generation warning

* Add Resequencer Node documentation

Closes #447 .

* Address review comment

Merging now because this affects documentation only, and this allows progress to continue on other pull requests.

* Handle timing types in elements without crashing (#516)

* Fix #492

* Define the type of `ebuttm:documentStartOfProgramme` correctly.
* When processing a timing type outside of the context of an attribute, bypass the timebase validation.

This is possibly a temporary hack, since element timing validation might be needed one day, but since we don't do anything with metadata elements, this isn't a disaster (yet).

* Add tests

Checks that including a valid time in a `ebuttm:documentStartOfProgramme` element does not cause a processing or validation error.

* Issue 507 timedelta (#508)

* Correctly interpret decimal fractions of seconds

Fixes #507 by interpreting decimal fractions of seconds as milliseconds and adding tests to verify this behaviour for 3 and 4 decimal places.

Removes `_int_or_none` which doesn't seem to be needed

* Specify incompatible attributes (#518)

Fixes #514.

* Resequencer: allow to immediately issue documents (#510)

So far the resequencer starts to regularly issue documents only after
the first EBU-TT Live document has been received. However for some use
cases this might be inconvenient as an active document must exist at all
times e.g. when creating segments for an MPEG-DASH stream. The reason
for documents being issued only after the first received EBU-TT Live
document is that certain parameters of that document are used for
initialisation.

This commit adds a new configuration parameter that specifies a document
which will be used for initialisation (instead of the first received
EBU-TT Live document). Therefore the resequencer will immediately (be
able to) start issuing documents after its creation.

Closes #505.

* Resequencer: fix sequence number 1 if no content (#509)

While the EBUTT3Splicer is provided with the current sequence number
stored/incremented by the Resequencer, the `create_compatible_document`
method so far refers to the sequence number stored internally by
EBUTT3DocumentSequence instead. This leads to a sequence number of 1 in
the issued output document if no subtitle content is available.

To align with the correct (strictly monotonic increasing) sequence
number, always use the one stored by the Resequencer.

Fixes #502.

* Up version to 2.1.2

docs/source/conf.py

@@ -77,9 +77,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = u'2.1.1'
+version = u'2.1.2'
 # The full version, including alpha/beta/rc tags.
-release = u'2.1.1'
+release = u'2.1.2'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

docs/source/configurator.rst

@@ -99,6 +99,7 @@ Node type dependent options for [nodeN] : ::
    ├─sequence_identifier : sequence identifier, default "re-sequencer"
    ├─segment_length : duration of each output segment in seconds, default 2
    ├─begin_output : ["immediate" (default) | {begin time} ] the time at which the first output segment should begin.
+   ├─init_document : File path to a document used to set initial parameters and afterwards start issuing documents (instead of awaiting the first received document), default None
    ├─discard : whether to discard content that has been encoded, default True
    └─clock
      └─type : ["local" (default) | "auto" | "clock"]

docs/source/deduplication.rst

@@ -13,7 +13,7 @@ After copying ``styling`` and ``layout`` into a ``list()`` and setting them up f
 Because ``style`` and ``region`` elements can have ``style`` attributes, these
 are deduplicated first. At this stage, it's possible that where two identical elements
 that differed only in their style references, these may end up looking the same.
- Each element is then passed through the
+Each element is then passed through the
 :py:class:`ebu_tt_live.node.deduplicator.ComparableElement` class, which processes
 each attribute, omitting the ``xml:id`` and using the
 :py:func:`ebu_tt_live.node.deduplicator.ReplaceNone` function to replace empty

docs/source/ebu_tt_live.node.rst

@@ -74,7 +74,7 @@ node Package
     :show-inheritance:
 
 :mod:`deduplicator` Module
-----------------------
+--------------------------
 
 .. automodule:: ebu_tt_live.node.deduplicator
     :members:

docs/source/scripts_and_their_functions.rst

@@ -67,11 +67,11 @@ consumer is listening. See detailed instructions here:
 Distributor
 -----------
 This script mimics a distribution node. To see it forwarding documents from the
-Simple Producer the the Simple Consumer using Websocket, run ``ebu-run
+Simple Producer to the Simple Consumer using Websocket, run ``ebu-run
 --admin.conf=ebu_tt_live/examples/config/sproducer_dist_sconsumer_ws.json``. A
 more interesting scenario is distributing documents from the User Input Producer
 to two consumer nodes: ``ebu-run
---admin.conf=ebu_tt_live/examples/config/user_input_producer_dist_consumers``.
+--admin.conf=ebu_tt_live/examples/config/user_input_producer_dist_consumers.json``.
 
 Like the Simple Producer, the Distributor can also save the documents it
 receives to the file system. To do that, create you own configuration file as
@@ -103,10 +103,46 @@ example ``ebu-run --admin.conf=ebu_tt_live/examples/config/buffer_delay.json``
 DeDuplicator Node
 -----------------
 This node addresses instances where ``style`` and ``region`` elements and
-attributes are duplicated.
+attributes are duplicated, which can occur for example when sequences are
+resequenced.
 For the default configuration of the node, see:
 ``ebu-run --admin.conf=ebu_tt_live/examples/config/deduplicator_fs.json``
 
+Resequencer Node
+----------------
+This node consumes documents from one sequence and
+creates a new sequence of documents based on the content in that input sequence,
+where every document in the output sequence has the same duration. The
+resequencer repeatedly extracts and then outputs a document of the specified
+duration, then waits for a period equal to that 
+duration before extracting the next document. It can be configured to begin
+extracting the first document immediately when it is run, or to wait until a
+specific time until extracting the first document. 
+
+The resequencer is
+particularly useful upstream of an EBU-TT-D Encoder, to generate segmented
+EBU-TT-D, for example prior to wrapping in fragmented MPEG-4 and serving
+with a DASH or HLS manifest; those onward steps are not part of this project.
+This pattern effectively converts an asynchronous stream of input documents
+into something that can be delivered synchronously downstream, which is
+useful for distribution to media players.
+
+Note that the resequencer output can contain duplicated ``style`` and ``region``
+elements. These can be cleaned up by passing the output to a DeDuplicator
+node before downstream encoding to other formats.
+
+In general the resequencer does not begin emitting any documents until it has received
+at least one input document. To immediately start to emit documents an
+initial document can be configured. Necessary initial parameters like
+language or sequence ID are retrieved from that document.
+
+Note that the resequencer accepts only input documents which all have the
+same sequence ID. This sequence ID is determined by the first received input
+document (or the configured initial document instead, if applicable).
+
+Use ``ebu-run`` to start
+this script, for example ``ebu-run --admin.conf=ebu_tt_live/examples/config/sproducer_resequencer_direct_ebuttd_encoder_fs.json``
+
 Retiming Delay Node
 -------------------
 This script modifies the times within each Document and issues them without

ebu_tt_live/bindings/_ebuttdt.py

@@ -37,13 +37,6 @@ class _TimedeltaBindingMixin(object):
         'end': []
     }
 
-    @classmethod
-    def _int_or_none(cls, value):
-        try:
-            return int(value)
-        except TypeError:
-            return 0
-
     @classmethod
     def compatible_timebases(cls):
         return cls._compatible_timebases
@@ -70,20 +63,25 @@ def _ConvertArguments_vx(cls, args, kw):
             # This means we are in XML parsing context. There should be a timeBase and a timing_attribute_name in the
             # context object.
             time_base = context['timeBase']
-            timing_att_name = context['timing_attribute_name']
-            if time_base not in cls._compatible_timebases[timing_att_name]:
-                log.debug(ERR_SEMANTIC_VALIDATION_TIMING_TYPE.format(
-                    attr_name=timing_att_name,
-                    attr_type=cls,
-                    attr_value=args,
-                    time_base=time_base
-                ))
-                raise pyxb.SimpleTypeValueError(ERR_SEMANTIC_VALIDATION_TIMING_TYPE.format(
-                    attr_name=timing_att_name,
-                    attr_type=cls,
-                    attr_value=args,
-                    time_base=time_base
-                ))
+            # It is possible for a timing type to exist as the value of an element not an attribute,
+            # in which case no timing_attribute_name is in the context; in that case don't attempt
+            # to validate the data against a timebase. At the moment this only affects the
+            # documentStartOfProgramme metadata element.
+            if 'timing_attribute_name' in context:
+                timing_att_name = context['timing_attribute_name']
+                if time_base not in cls._compatible_timebases[timing_att_name]:
+                    log.debug(ERR_SEMANTIC_VALIDATION_TIMING_TYPE.format(
+                        attr_name=timing_att_name,
+                        attr_type=cls,
+                        attr_value=args,
+                        time_base=time_base
+                    ))
+                    raise pyxb.SimpleTypeValueError(ERR_SEMANTIC_VALIDATION_TIMING_TYPE.format(
+                        attr_name=timing_att_name,
+                        attr_type=cls,
+                        attr_value=args,
+                        time_base=time_base
+                    ))
         for item in args:
             if isinstance(item, timedelta):
                 result.append(cls.from_timedelta(item))
@@ -319,11 +317,12 @@ def as_timedelta(cls, instance):
         :param instance:
         :return:
         """
-        hours, minutes, seconds, milliseconds = map(
-            lambda x: cls._int_or_none(x),
-            cls._groups_regex.match(instance).groups()
-        )
-        return timedelta(hours=hours, minutes=minutes, seconds=seconds, milliseconds=milliseconds)
+        hours_str, minutes_str, seconds_str, seconds_fraction_str = [x for x in cls._groups_regex.match(instance).groups()]
+        milliseconds = seconds_fraction_str and float('0.' + seconds_fraction_str) * 1000 or 0
+        return timedelta(hours=int(hours_str), 
+            minutes=int(minutes_str), 
+            seconds=int(seconds_str), 
+            milliseconds=milliseconds)
 
     @classmethod
     def from_timedelta(cls, instance):
@@ -370,11 +369,13 @@ def as_timedelta(cls, instance):
         :param instance:
         :return:
         """
-        hours, minutes, seconds, milliseconds = map(
-            lambda x: cls._int_or_none(x),
-            cls._groups_regex.match(instance).groups()
-        )
-        return timedelta(hours=hours, minutes=minutes, seconds=seconds, milliseconds=milliseconds)
+        hours_str, minutes_str, seconds_str, seconds_fraction_str = [x for x in cls._groups_regex.match(instance).groups()]
+        milliseconds = seconds_fraction_str and float('0.' + seconds_fraction_str) * 1000 or 0
+        return timedelta(hours=int(hours_str), 
+            minutes=int(minutes_str), 
+            seconds=int(seconds_str), 
+            milliseconds=milliseconds)
+
 
     @classmethod
     def from_timedelta(cls, instance):

ebu_tt_live/bindings/test/test_times.py

@@ -0,0 +1,93 @@
+
+from unittest import TestCase
+from unittest import skip, SkipTest
+from datetime import timedelta
+from ebu_tt_live.bindings import ebuttdt
+from pyxb import SimpleTypeValueError, SimpleFacetValueError
+
+class TestTimecountTimingType(TestCase):
+
+    _test_cases = {
+        '1.3s': timedelta(seconds = 1, milliseconds = 300),
+        '1.03s': timedelta(seconds = 1, milliseconds = 30),
+        '1.30s': timedelta(seconds = 1, milliseconds = 300),
+        '1.003s': timedelta(seconds = 1, milliseconds = 3),
+        '1.030s': timedelta(seconds = 1, milliseconds = 30),
+        '1.300s': timedelta(seconds = 1, milliseconds = 300),
+        '1.3456s': timedelta(seconds = 1, milliseconds = 345.6),
+        '1s': timedelta(seconds = 1),
+        '1.3m': timedelta(minutes = 1, seconds = 18),
+        '1.03m': timedelta(minutes = 1, seconds = 1, milliseconds = 800),
+        '1.30m': timedelta(minutes = 1, seconds = 18),
+        '1.003m': timedelta(minutes = 1, milliseconds = 180),
+        '1.030m': timedelta(minutes = 1, seconds = 1, milliseconds = 800),
+        '1.300m': timedelta(minutes = 1, seconds = 18),
+        '1m': timedelta(minutes = 1),
+        '1.3h': timedelta(hours = 1, minutes = 18),
+        '1.03h': timedelta(hours = 1, minutes = 1, seconds = 48),
+        '1.30h': timedelta(hours = 1, minutes = 18),
+        '1.003h': timedelta(hours = 1, seconds=10, milliseconds = 800),
+        '1.030h': timedelta(hours = 1, minutes = 1, seconds = 48),
+        '1.300h': timedelta(hours = 1, minutes = 18),
+        '1h': timedelta(hours = 1),
+        '1.3ms': timedelta(milliseconds = 1, microseconds = 300),
+        '1.03ms': timedelta(milliseconds = 1, microseconds = 30),
+        '1.30ms': timedelta(milliseconds = 1, microseconds = 300),
+        '1.003ms': timedelta(milliseconds = 1, microseconds = 3),
+        '1.030ms': timedelta(milliseconds = 1, microseconds = 30),
+        '1.300ms': timedelta(milliseconds = 1, microseconds = 300),
+        '1ms': timedelta(milliseconds = 1),
+    }
+
+    _type_class = ebuttdt.TimecountTimingType
+
+    def test_as_timedelta(self):
+        for t, td in self._test_cases.items() :
+            test_instance = self._type_class(t)
+            self.assertEqual(test_instance.timedelta, td)
+            assert test_instance == t
+
+
+
+class TestFullClockTimingType(TestCase):
+
+    _test_cases = {
+        '111:22:33': timedelta(hours = 111, minutes = 22, seconds = 33),
+        '111:22:33.4': timedelta(hours = 111, minutes = 22, seconds = 33, milliseconds = 400),
+        '111:22:33.04': timedelta(hours = 111, minutes = 22, seconds = 33, milliseconds = 40),
+        '111:22:33.40': timedelta(hours = 111, minutes = 22, seconds = 33, milliseconds = 400),
+        '111:22:33.400': timedelta(hours = 111, minutes = 22, seconds = 33, milliseconds = 400),
+        '111:22:33.040': timedelta(hours = 111, minutes = 22, seconds = 33, milliseconds = 40),
+        '111:22:33.004': timedelta(hours = 111, minutes = 22, seconds = 33, milliseconds = 4),
+        '111:22:33.0045': timedelta(hours = 111, minutes = 22, seconds = 33, milliseconds = 4.5),
+    }
+
+    _type_class = ebuttdt.FullClockTimingType
+
+    def test_as_timedelta(self):
+        for t, td in self._test_cases.items() :
+            test_instance = self._type_class(t)
+            self.assertEqual(test_instance.timedelta, td)
+            assert test_instance == t
+
+
+class TestLimitedClockTimingType(TestCase):
+
+    _test_cases = {
+        '11:22:33': timedelta(hours = 11, minutes = 22, seconds = 33),
+        '11:22:33.4': timedelta(hours = 11, minutes = 22, seconds = 33, milliseconds = 400),
+        '11:22:33.04': timedelta(hours = 11, minutes = 22, seconds = 33, milliseconds = 40),
+        '11:22:33.40': timedelta(hours = 11, minutes = 22, seconds = 33, milliseconds = 400),
+        '11:22:33.400': timedelta(hours = 11, minutes = 22, seconds = 33, milliseconds = 400),
+        '11:22:33.040': timedelta(hours = 11, minutes = 22, seconds = 33, milliseconds = 40),
+        '11:22:33.004': timedelta(hours = 11, minutes = 22, seconds = 33, milliseconds = 4),
+        '11:22:33.0045': timedelta(hours = 11, minutes = 22, seconds = 33, milliseconds = 4.5),
+    }
+
+    _type_class = ebuttdt.LimitedClockTimingType
+
+    def test_as_timedelta(self):
+        for t, td in self._test_cases.items() :
+            test_instance = self._type_class(t)
+            self.assertEqual(test_instance.timedelta, td)
+            assert test_instance == t

ebu_tt_live/bindings/validation/presentation.py

@@ -43,14 +43,14 @@ class StyledElementMixin(object):
     def _semantic_collect_applicable_styles(self, dataset, style_type, parent_binding, defer_font_size=False,
                                             extra_referenced_styles=None):
         """
-        This function identifies the styling dependdncy chain for the styled element in question.
+        This function identifies the styling dependency chain for the styled element in question.
 
         :param dataset: Semantic dataset
-        :param style_type: the style_type to be used in the process (there are different style types for EBU-TT D and
-        live).
+        :param style_type: the style_type to be used in the process (there are different style types for EBU-TT D and live).
         :param parent_binding: The immediate parent of the styled element in the document structure
         :param defer_font_size: If True then fontsize can stay percentage in case it could not be calculated
         :param extra_referenced_styles: Used by region to inject its extra style attributes
+
         :return:
         """
 

ebu_tt_live/config/backend.py

@@ -137,8 +137,6 @@ def _ws_create_server_factory(self, listen, producer=None, consumer=None):
 
     def _ws_create_client_factories(self, connect, producer=None, consumer=None, proxy=None):
         factory_args = {}
-        if proxy:
-            factory_args.update({'host': proxy.host, 'port': proxy.port})
         for dst in connect:
             client_factory = self._websocket.BroadcastClientFactory(
                 url=dst.geturl(),
@@ -147,6 +145,8 @@ def _ws_create_client_factories(self, connect, producer=None, consumer=None, pro
                 **factory_args
             )
             client_factory.protocol = self._websocket.BroadcastClientProtocol
+            client_factory.proxy = proxy
+
             client_factory.connect()
 
     def ws_backend_producer(self, custom_producer, listen=None, connect=None, proxy=None):

ebu_tt_live/config/carriage.py

@@ -2,7 +2,6 @@
 from ebu_tt_live.carriage.direct import DirectCarriageImpl
 from ebu_tt_live.carriage.websocket import WebsocketProducerCarriage, WebsocketConsumerCarriage
 from ebu_tt_live.carriage import filesystem
-from ebu_tt_live.utils import HTTPProxyConfig
 from ebu_tt_live.strings import ERR_CONF_PROXY_CONF_VALUE, ERR_NO_SUCH_COMPONENT
 from ebu_tt_live.errors import ConfigurationError
 from ebu_tt_live.strings import CFG_FILENAME_PATTERN, CFG_MESSAGE_PATTERN
@@ -134,10 +133,7 @@ def parse_proxy_address(value):
     match = proxy_regex.match(value)
     if match:
         # Ignoring the protocol part for now as it is only a http proxy
-        result = HTTPProxyConfig(
-            host=match.group('host'),
-            port=int(match.group('port'))
-        )
+        result = {u'host': match.group('host'), u'port': int(match.group('port'))}
     elif value:
         # In this case something was provided that isn't a falsy value but the parsing failed.
         raise ConfigurationError(

ebu_tt_live/config/node.py

@@ -102,6 +102,7 @@ class ReSequencer(ProducerMixin, ConsumerMixin, NodeBase):
     required_config.add_option('segment_length', default=2.0)
     required_config.clock = Namespace()
     required_config.clock.add_option('type', default='local', from_string_converter=get_clock)
+    required_config.add_option('init_document', default=None)
     required_config.add_option('discard', default=True)
     required_config.add_option(
         'begin_output',
@@ -128,6 +129,7 @@ def _create_component(self, config=None):
         self.component = processing_node.ReSequencer(
             node_id=self.config.id,
             reference_clock=self._clock.component,
+            init_document=self.config.init_document,
             discard=self.config.discard,
             segment_length=self.config.segment_length,
             sequence_identifier=self.config.sequence_identifier

ebu_tt_live/documents/ebutt3.py

@@ -642,12 +642,12 @@ def _check_document_compatibility(self, document):
         if self.sequence_identifier != document.sequence_identifier or \
                 self._reference_clock.time_base != document.time_base:
             raise IncompatibleSequenceError(
-                ERR_DOCUMENT_NOT_COMPATIBLE
+                ERR_DOCUMENT_NOT_COMPATIBLE.format('ebuttp:sequenceIdentifier or ttp:timeBase')
             )
         if self._reference_clock.time_base == 'clock':
             if self._reference_clock.clock_mode != document.clock_mode:
                 raise IncompatibleSequenceError(
-                    ERR_DOCUMENT_NOT_COMPATIBLE
+                    ERR_DOCUMENT_NOT_COMPATIBLE.format('ttp:clockMode')
                 )
         existing_document = None
         try:
@@ -706,7 +706,7 @@ def create_compatible_document(self, *args, **kwargs):
             clock_mode=self._reference_clock.clock_mode,
             sequence_identifier=self._sequence_identifier,
             authors_group_identifier=self.authors_group_identifier,
-            sequence_number=self._last_sequence_number,
+            sequence_number=kwargs.get('sequence_number', self._last_sequence_number),
             lang=self._lang
         )
 
@@ -992,15 +992,16 @@ def extract_segment(self, begin=None, end=None, sequence_number=None, discard=Fa
 
             begin = doc_ending
 
+        current_sequence_number = sequence_number is not None and sequence_number or 1
         if not document_segments:
             # TODO: This is good question what now? no documents found for range
-            document = self.create_compatible_document()
+            document = self.create_compatible_document(sequence_number=current_sequence_number)
             # comp_doc.set_begin(begin)
             # comp_doc.set_end(end)
         else:
             splicer = EBUTT3Splicer(
                 sequence_identifier='{}_resegmented'.format(self.sequence_identifier),
-                sequence_number=sequence_number is not None and sequence_number or 1,
+                sequence_number=current_sequence_number,
                 document_segments=document_segments
             )
             document = EBUTT3Document.create_from_raw_binding(splicer.spliced_document)