Release/2.1.2 (ebu#522)

* Add service_identity module dependency

Removes a test warning, improves twisted's ability to verify certificates.

* Fix ebu#474

Use relative links to templates.
Put message payload into the UIP display so that the receiver can read it without having to open the message XML up.

* Remove requirement for pytest-capturelog, removed from

No longer needed because it has been merged into the core.

* Refactor proxy handling for client

Track a change to proxy handling in Autobahn / twisted for ebu#477. There's probably a more elegant way, but this works for now... Lacking a test, but have verified it locally.

* More elegant fix for proxy

We don't need the HTTPProxyConfig object at all anymore, just a `dict` will do.

* Don't call fixtures directly

Closes ebu#490.

Calling fixtures directly is deprecated, this solution described at
https://docs.pytest.org/en/latest/deprecations.html#calling-fixtures-directly
seems to work, creating a named fixture rather than defining the "then"
step as a fixture directly.

* Restrict autobahn to versions before 18

Temporarily fixes ebu#496, prior to addressing the problem properly anyway.

* Restrict autobahn to versions before 18

Temporarily fixes ebu#496, prior to addressing the problem properly anyway.

* Issue 447 resequencer documentation (ebu#515)

* Fix indentation warning

* Fix underline length warning

* Fix document generation warning

* Add Resequencer Node documentation

Closes ebu#447 .

* Address review comment

Merging now because this affects documentation only, and this allows progress to continue on other pull requests.

* Handle timing types in elements without crashing (ebu#516)

* Fix ebu#492

* Define the type of `ebuttm:documentStartOfProgramme` correctly.
* When processing a timing type outside of the context of an attribute, bypass the timebase validation.

This is possibly a temporary hack, since element timing validation might be needed one day, but since we don't do anything with metadata elements, this isn't a disaster (yet).

* Add tests

Checks that including a valid time in a `ebuttm:documentStartOfProgramme` element does not cause a processing or validation error.

* Issue 507 timedelta (ebu#508)

* Correctly interpret decimal fractions of seconds

Fixes ebu#507 by interpreting decimal fractions of seconds as milliseconds and adding tests to verify this behaviour for 3 and 4 decimal places.

Removes `_int_or_none` which doesn't seem to be needed

* Specify incompatible attributes (ebu#518)

Fixes ebu#514.

* Resequencer: allow to immediately issue documents (ebu#510)

So far the resequencer starts to regularly issue documents only after
the first EBU-TT Live document has been received. However for some use
cases this might be inconvenient as an active document must exist at all
times e.g. when creating segments for an MPEG-DASH stream. The reason
for documents being issued only after the first received EBU-TT Live
document is that certain parameters of that document are used for
initialisation.

This commit adds a new configuration parameter that specifies a document
which will be used for initialisation (instead of the first received
EBU-TT Live document). Therefore the resequencer will immediately (be
able to) start issuing documents after its creation.

Closes ebu#505.

* Resequencer: fix sequence number 1 if no content (ebu#509)

While the EBUTT3Splicer is provided with the current sequence number
stored/incremented by the Resequencer, the `create_compatible_document`
method so far refers to the sequence number stored internally by
EBUTT3DocumentSequence instead. This leads to a sequence number of 1 in
the issued output document if no subtitle content is available.

To align with the correct (strictly monotonic increasing) sequence
number, always use the one stored by the Resequencer.

Fixes ebu#502.

* Up version to 2.1.2

ebu_tt_live/bindings/validation/presentation.py

@@ -43,7 +43,7 @@ class StyledElementMixin(object):
     def _semantic_collect_applicable_styles(self, dataset, style_type, parent_binding, defer_font_size=False,
                                             extra_referenced_styles=None):
         """
-        This function identifies the styling dependdncy chain for the styled element in question.
+        This function identifies the styling dependency chain for the styled element in question.
 
         :param dataset: Semantic dataset
         :param style_type: the style_type to be used in the process (there are different style types for EBU-TT D and live).

ebu_tt_live/config/backend.py

@@ -137,8 +137,6 @@ def _ws_create_server_factory(self, listen, producer=None, consumer=None):
 
     def _ws_create_client_factories(self, connect, producer=None, consumer=None, proxy=None):
         factory_args = {}
-        if proxy:
-            factory_args.update({'host': proxy.host, 'port': proxy.port})
         for dst in connect:
             client_factory = self._websocket.BroadcastClientFactory(
                 url=dst.geturl(),
@@ -147,6 +145,8 @@ def _ws_create_client_factories(self, connect, producer=None, consumer=None, pro
                 **factory_args
             )
             client_factory.protocol = self._websocket.BroadcastClientProtocol
+            client_factory.proxy = proxy
+
             client_factory.connect()
 
     def ws_backend_producer(self, custom_producer, listen=None, connect=None, proxy=None):

ebu_tt_live/config/carriage.py

@@ -2,7 +2,6 @@
 from ebu_tt_live.carriage.direct import DirectCarriageImpl
 from ebu_tt_live.carriage.websocket import WebsocketProducerCarriage, WebsocketConsumerCarriage
 from ebu_tt_live.carriage import filesystem
-from ebu_tt_live.utils import HTTPProxyConfig
 from ebu_tt_live.strings import ERR_CONF_PROXY_CONF_VALUE, ERR_NO_SUCH_COMPONENT
 from ebu_tt_live.errors import ConfigurationError
 from ebu_tt_live.strings import CFG_FILENAME_PATTERN, CFG_MESSAGE_PATTERN
@@ -134,10 +133,7 @@ def parse_proxy_address(value):
     match = proxy_regex.match(value)
     if match:
         # Ignoring the protocol part for now as it is only a http proxy
-        result = HTTPProxyConfig(
-            host=match.group('host'),
-            port=int(match.group('port'))
-        )
+        result = {u'host': match.group('host'), u'port': int(match.group('port'))}
     elif value:
         # In this case something was provided that isn't a falsy value but the parsing failed.
         raise ConfigurationError(

ebu_tt_live/config/node.py

@@ -102,6 +102,7 @@ class ReSequencer(ProducerMixin, ConsumerMixin, NodeBase):
     required_config.add_option('segment_length', default=2.0)
     required_config.clock = Namespace()
     required_config.clock.add_option('type', default='local', from_string_converter=get_clock)
+    required_config.add_option('init_document', default=None)
     required_config.add_option('discard', default=True)
     required_config.add_option(
         'begin_output',
@@ -128,6 +129,7 @@ def _create_component(self, config=None):
         self.component = processing_node.ReSequencer(
             node_id=self.config.id,
             reference_clock=self._clock.component,
+            init_document=self.config.init_document,
             discard=self.config.discard,
             segment_length=self.config.segment_length,
             sequence_identifier=self.config.sequence_identifier

ebu_tt_live/documents/ebutt3.py

@@ -544,12 +544,12 @@ def _check_document_compatibility(self, document):
         if self.sequence_identifier != document.sequence_identifier or \
                 self._reference_clock.time_base != document.time_base:
             raise IncompatibleSequenceError(
-                ERR_DOCUMENT_NOT_COMPATIBLE
+                ERR_DOCUMENT_NOT_COMPATIBLE.format('ebuttp:sequenceIdentifier or ttp:timeBase')
             )
         if self._reference_clock.time_base == 'clock':
             if self._reference_clock.clock_mode != document.clock_mode:
                 raise IncompatibleSequenceError(
-                    ERR_DOCUMENT_NOT_COMPATIBLE
+                    ERR_DOCUMENT_NOT_COMPATIBLE.format('ttp:clockMode')
                 )
         existing_document = None
         try:
@@ -608,7 +608,7 @@ def create_compatible_document(self, *args, **kwargs):
             clock_mode=self._reference_clock.clock_mode,
             sequence_identifier=self._sequence_identifier,
             authors_group_identifier=self.authors_group_identifier,
-            sequence_number=self._last_sequence_number,
+            sequence_number=kwargs.get('sequence_number', self._last_sequence_number),
             lang=self._lang
         )
 
@@ -894,15 +894,16 @@ def extract_segment(self, begin=None, end=None, sequence_number=None, discard=Fa
 
             begin = doc_ending
 
+        current_sequence_number = sequence_number is not None and sequence_number or 1
         if not document_segments:
             # TODO: This is good question what now? no documents found for range
-            document = self.create_compatible_document()
+            document = self.create_compatible_document(sequence_number=current_sequence_number)
             # comp_doc.set_begin(begin)
             # comp_doc.set_end(end)
         else:
             splicer = EBUTT3Splicer(
                 sequence_identifier='{}_resegmented'.format(self.sequence_identifier),
-                sequence_number=sequence_number is not None and sequence_number or 1,
+                sequence_number=current_sequence_number,
                 document_segments=document_segments
             )
             document = EBUTT3Document.create_from_raw_binding(splicer.spliced_document)

ebu_tt_live/node/consumer.py

@@ -35,9 +35,7 @@ def process_document(self, document, **kwargs):
                 log.info('Creating document sequence from first document {}'.format(
                     document
                 ))
-                self._sequence = EBUTT3DocumentSequence.create_from_document(document, verbose=self._verbose)
-                if self._reference_clock is None:
-                    self._reference_clock = self._sequence.reference_clock
+                self.create_sequence_from_document(document)
             if document.availability_time is None:
                 document.availability_time = self._reference_clock.get_time()
 
@@ -56,6 +54,11 @@ def process_document(self, document, **kwargs):
                     )
                 )
 
+    def create_sequence_from_document(self, document):
+        self._sequence = EBUTT3DocumentSequence.create_from_document(document, verbose=self._verbose)
+        if self._reference_clock is None:
+            self._reference_clock = self._sequence.reference_clock
+
     @property
     def reference_clock(self):
         return self._reference_clock
@@ -77,7 +80,7 @@ class ReSequencer(AbstractProducerNode, SimpleConsumer):
     _provides = EBUTT3Document
 
     def __init__(self, node_id, reference_clock, segment_length, discard, sequence_identifier,
-                 consumer_carriage=None, producer_carriage=None, **kwargs):
+                 consumer_carriage=None, producer_carriage=None, init_document=None, **kwargs):
         super(ReSequencer, self).__init__(
             node_id=node_id,
             consumer_carriage=consumer_carriage,
@@ -91,6 +94,16 @@ def __init__(self, node_id, reference_clock, segment_length, discard, sequence_i
         self._segment_counter = 1
         self._sequence_identifier = sequence_identifier
         self._discard = discard
+
+        if init_document is not None:
+            # Create sequence from init document, in order to immediately start document output
+            log.info('Creating document sequence from init document {}'.format(
+                init_document
+            ))
+            with open(init_document, 'r') as xml_file:
+                xml_content = xml_file.read()
+            xml_doc = EBUTT3Document.create_from_xml(xml_content)
+            self.create_sequence_from_document(xml_doc)
 
     @property
     def last_segment_end(self):

ebu_tt_live/ui/user_input_producer/index.html

@@ -174,10 +174,11 @@ <h3>This is the User Input Producer (UIP).</h3>
                <div class="doc-item-template doc-item">
                   <span>Seq.Num.: </span><span class="seqNum-value"></span><span>  --  </span>
                   <span>Au.G.Id.: </span><span class="autGID-value"></span><span>  --  </span>
-                  <span>Au.G.C.Token: </span><snap class="autGCT-value"></snap>
+                  <span>Au.G.C.Token: </span><span class="autGCT-value"></span>
                </div>
                <div class="message-item-template message-item">
                   <span>Message from sender: </span><span class="sender-value"></span>
+                  <span> Payload: </span><span class="payload"></span>
                </div>
             </div>
             <div id="result-view-div">

ebu_tt_live/ui/user_input_producer/uip.js

@@ -178,6 +178,11 @@ $(document).ready(function() {
                 'urn:ebu:tt:livemessage', 'sender'
             )[0].firstChild.nodeValue
         );
+        clonedElement.find('.payload').text(
+          message.getElementsByTagNameNS(
+            'urn:ebu:tt:livemessage', 'payload'
+          )[0].firstChild.nodeValue
+        );
 
         clonedElement.removeClass('message-item-template');
         clonedElement.addClass('result-list-item');
@@ -512,7 +517,7 @@ $(document).ready(function() {
 
     function renderHandoverMessageTemplate(template_data) {
         var rendered_document = nunjucks.render(
-          'ebu_tt_live/ui/user_input_producer/template/live_message_template.xml',
+          'template/live_message_template.xml',
           template_data
         );
         return rendered_document;
@@ -572,7 +577,7 @@ $(document).ready(function() {
             var timeout = computeScheduledSendTimeout(media_offset);
             var template_data = createTemplateDict();
             var rendered_document = nunjucks.render(
-              'ebu_tt_live/ui/user_input_producer/template/user_input_producer_template.xml',
+              'template/user_input_producer_template.xml',
               template_data
             );
 
@@ -588,7 +593,7 @@ $(document).ready(function() {
 
     function asyncSubmit() {
         var template_data = createTemplateDict();
-        var rendered_document = nunjucks.render('ebu_tt_live/ui/user_input_producer/template/user_input_producer_template.xml', template_data);
+        var rendered_document = nunjucks.render('template/user_input_producer_template.xml', template_data);
         renderSendDocument(rendered_document);
         sequence_numbers[sequence_identifier] += 1;
         localStorage.sequence_numbers = JSON.stringify(sequence_numbers);
@@ -600,7 +605,7 @@ $(document).ready(function() {
         // If doc in not null then it was already rendered (needed for scheduled times : the document
         // has to be rendered when the sending is scheduled not at effective sending time.
         if (doc == null) {
-          rendered_document = nunjucks.render('ebu_tt_live/ui/user_input_producer/template/user_input_producer_template.xml', template_data);
+          rendered_document = nunjucks.render('template/user_input_producer_template.xml', template_data);
         } else {
           rendered_document = doc;
         }

ebu_tt_live/utils.py

@@ -360,8 +360,6 @@ def __call__(cls, *args, **kwargs):
         instance = super(AutoRegisteringABCMeta, cls).__call__(*args, **kwargs)
         return instance
 
-HTTPProxyConfig = collections.namedtuple('HTTPProxyConfig', ['host', 'port'])
-
 
 # The following section is taken from https://github.com/django/django/blob/master/django/test/utils.py
 # This is a relatively simple XML comparator implementation based on Python's minidom library.
@@ -469,4 +467,4 @@ def first_node(document):
     want_root = first_node(parseString(want))
     got_root = first_node(parseString(got))
 
-    return check_element(want_root, got_root)
+    return check_element(want_root, got_root)