Release/2.1.2 (ebu#522)

* Add service_identity module dependency

Removes a test warning, improves twisted's ability to verify certificates.

* Fix ebu#474

Use relative links to templates.
Put message payload into the UIP display so that the receiver can read it without having to open the message XML up.

* Remove requirement for pytest-capturelog, removed from

No longer needed because it has been merged into the core.

* Refactor proxy handling for client

Track a change to proxy handling in Autobahn / twisted for ebu#477. There's probably a more elegant way, but this works for now... Lacking a test, but have verified it locally.

* More elegant fix for proxy

We don't need the HTTPProxyConfig object at all anymore, just a `dict` will do.

* Don't call fixtures directly

Closes ebu#490.

Calling fixtures directly is deprecated, this solution described at
https://docs.pytest.org/en/latest/deprecations.html#calling-fixtures-directly
seems to work, creating a named fixture rather than defining the "then"
step as a fixture directly.

* Restrict autobahn to versions before 18

Temporarily fixes ebu#496, prior to addressing the problem properly anyway.

* Restrict autobahn to versions before 18

Temporarily fixes ebu#496, prior to addressing the problem properly anyway.

* Issue 447 resequencer documentation (ebu#515)

* Fix indentation warning

* Fix underline length warning

* Fix document generation warning

* Add Resequencer Node documentation

Closes ebu#447 .

* Address review comment

Merging now because this affects documentation only, and this allows progress to continue on other pull requests.

* Handle timing types in elements without crashing (ebu#516)

* Fix ebu#492

* Define the type of `ebuttm:documentStartOfProgramme` correctly.
* When processing a timing type outside of the context of an attribute, bypass the timebase validation.

This is possibly a temporary hack, since element timing validation might be needed one day, but since we don't do anything with metadata elements, this isn't a disaster (yet).

* Add tests

Checks that including a valid time in a `ebuttm:documentStartOfProgramme` element does not cause a processing or validation error.

* Issue 507 timedelta (ebu#508)

* Correctly interpret decimal fractions of seconds

Fixes ebu#507 by interpreting decimal fractions of seconds as milliseconds and adding tests to verify this behaviour for 3 and 4 decimal places.

Removes `_int_or_none` which doesn't seem to be needed

* Specify incompatible attributes (ebu#518)

Fixes ebu#514.

* Resequencer: allow to immediately issue documents (ebu#510)

So far the resequencer starts to regularly issue documents only after
the first EBU-TT Live document has been received. However for some use
cases this might be inconvenient as an active document must exist at all
times e.g. when creating segments for an MPEG-DASH stream. The reason
for documents being issued only after the first received EBU-TT Live
document is that certain parameters of that document are used for
initialisation.

This commit adds a new configuration parameter that specifies a document
which will be used for initialisation (instead of the first received
EBU-TT Live document). Therefore the resequencer will immediately (be
able to) start issuing documents after its creation.

Closes ebu#505.

* Resequencer: fix sequence number 1 if no content (ebu#509)

While the EBUTT3Splicer is provided with the current sequence number
stored/incremented by the Resequencer, the `create_compatible_document`
method so far refers to the sequence number stored internally by
EBUTT3DocumentSequence instead. This leads to a sequence number of 1 in
the issued output document if no subtitle content is available.

To align with the correct (strictly monotonic increasing) sequence
number, always use the one stored by the Resequencer.

Fixes ebu#502.

* Up version to 2.1.2

docs/source/configurator.rst

@@ -101,8 +101,8 @@ Node type dependent options for [nodeN] : ::
    type="resequencer"
    ├─sequence_identifier : sequence identifier, default "re-sequencer"
    ├─segment_length : duration of each output segment in seconds, default 2
-   ├─begin_output : ["immediate" (default) | {begin time} ] the time at which the first output
-   │                segment should begin.
+   ├─begin_output : ["immediate" (default) | {begin time} ] the time at which the first output segment should begin.
+   ├─init_document : File path to a document used to set initial parameters and afterwards start issuing documents (instead of awaiting the first received document), default None
    ├─discard : whether to discard content that has been encoded, default True
    └─clock
      └─type : ["local" (default) | "auto" | "clock"]

docs/source/scripts_and_their_functions.rst

@@ -67,11 +67,11 @@ consumer is listening. See detailed instructions here:
 Distributor
 -----------
 This script mimics a distribution node. To see it forwarding documents from the
-Simple Producer the the Simple Consumer using Websocket, run ``ebu-run
+Simple Producer to the Simple Consumer using Websocket, run ``ebu-run
 --admin.conf=ebu_tt_live/examples/config/sproducer_dist_sconsumer_ws.json``. A
 more interesting scenario is distributing documents from the User Input Producer
 to two consumer nodes: ``ebu-run
---admin.conf=ebu_tt_live/examples/config/user_input_producer_dist_consumers``.
+--admin.conf=ebu_tt_live/examples/config/user_input_producer_dist_consumers.json``.
 
 Like the Simple Producer, the Distributor can also save the documents it
 receives to the file system. To do that, create you own configuration file as
@@ -103,7 +103,8 @@ example ``ebu-run --admin.conf=ebu_tt_live/examples/config/buffer_delay.json``
 DeDuplicator Node
 -----------------
 This node addresses instances where ``style`` and ``region`` elements and
-attributes are duplicated.
+attributes are duplicated, which can occur for example when sequences are
+resequenced.
 For the default configuration of the node, see:
 ``ebu-run --admin.conf=ebu_tt_live/examples/config/deduplicator_fs.json``
 
@@ -118,15 +119,40 @@ If nested ``div`` or ``span`` elements might be present in a document, the
 Denester node should be used to flatten them before passing them to the
 EBU-TT-D Encoder, because EBU-TT-D does not permit such nested elements.
 
-Resequencer
------------
-This node receives input documents from one sequence and periodically issues
-a document corresponding to a segment of time. Each time a document is
-issued the time of the next segment is incremented. For example this node
-can be used to extract a 5s chunk of subtitles every 5s.
-
-This node may be used upstream of the EBU-TT-D Encoder to generate an
-ongoing sequence of subtitle documents from a streaming source.
+Resequencer Node
+----------------
+This node consumes documents from one sequence and
+creates a new sequence of documents based on the content in that input sequence,
+where every document in the output sequence has the same duration. The
+resequencer repeatedly extracts and then outputs a document of the specified
+duration, then waits for a period equal to that 
+duration before extracting the next document. It can be configured to begin
+extracting the first document immediately when it is run, or to wait until a
+specific time until extracting the first document. 
+
+The resequencer is
+particularly useful upstream of an EBU-TT-D Encoder, to generate segmented
+EBU-TT-D, for example prior to wrapping in fragmented MPEG-4 and serving
+with a DASH or HLS manifest; those onward steps are not part of this project.
+This pattern effectively converts an asynchronous stream of input documents
+into something that can be delivered synchronously downstream, which is
+useful for distribution to media players.
+
+Note that the resequencer output can contain duplicated ``style`` and ``region``
+elements. These can be cleaned up by passing the output to a DeDuplicator
+node before downstream encoding to other formats.
+
+In general the resequencer does not begin emitting any documents until it has received
+at least one input document. To immediately start to emit documents an
+initial document can be configured. Necessary initial parameters like
+language or sequence ID are retrieved from that document.
+
+Note that the resequencer accepts only input documents which all have the
+same sequence ID. This sequence ID is determined by the first received input
+document (or the configured initial document instead, if applicable).
+
+Use ``ebu-run`` to start
+this script, for example ``ebu-run --admin.conf=ebu_tt_live/examples/config/sproducer_resequencer_direct_ebuttd_encoder_fs.json``
 
 Retiming Delay Node
 -------------------

ebu_tt_live/bindings/_ebuttdt.py

@@ -37,13 +37,6 @@ class _TimedeltaBindingMixin(object):
         'end': []
     }
 
-    @classmethod
-    def _int_or_none(cls, value):
-        try:
-            return int(value)
-        except TypeError:
-            return 0
-
     @classmethod
     def compatible_timebases(cls):
         return cls._compatible_timebases

ebu_tt_live/bindings/test/test_times.py

@@ -11,18 +11,31 @@ class TestTimecountTimingType(TestCase):
         '1.3s': timedelta(seconds = 1, milliseconds = 300),
         '1.03s': timedelta(seconds = 1, milliseconds = 30),
         '1.30s': timedelta(seconds = 1, milliseconds = 300),
+        '1.003s': timedelta(seconds = 1, milliseconds = 3),
+        '1.030s': timedelta(seconds = 1, milliseconds = 30),
+        '1.300s': timedelta(seconds = 1, milliseconds = 300),
+        '1.3456s': timedelta(seconds = 1, milliseconds = 345.6),
         '1s': timedelta(seconds = 1),
         '1.3m': timedelta(minutes = 1, seconds = 18),
         '1.03m': timedelta(minutes = 1, seconds = 1, milliseconds = 800),
         '1.30m': timedelta(minutes = 1, seconds = 18),
+        '1.003m': timedelta(minutes = 1, milliseconds = 180),
+        '1.030m': timedelta(minutes = 1, seconds = 1, milliseconds = 800),
+        '1.300m': timedelta(minutes = 1, seconds = 18),
         '1m': timedelta(minutes = 1),
         '1.3h': timedelta(hours = 1, minutes = 18),
         '1.03h': timedelta(hours = 1, minutes = 1, seconds = 48),
         '1.30h': timedelta(hours = 1, minutes = 18),
+        '1.003h': timedelta(hours = 1, seconds=10, milliseconds = 800),
+        '1.030h': timedelta(hours = 1, minutes = 1, seconds = 48),
+        '1.300h': timedelta(hours = 1, minutes = 18),
         '1h': timedelta(hours = 1),
         '1.3ms': timedelta(milliseconds = 1, microseconds = 300),
         '1.03ms': timedelta(milliseconds = 1, microseconds = 30),
         '1.30ms': timedelta(milliseconds = 1, microseconds = 300),
+        '1.003ms': timedelta(milliseconds = 1, microseconds = 3),
+        '1.030ms': timedelta(milliseconds = 1, microseconds = 30),
+        '1.300ms': timedelta(milliseconds = 1, microseconds = 300),
         '1ms': timedelta(milliseconds = 1),
     }
 
@@ -43,6 +56,10 @@ class TestFullClockTimingType(TestCase):
         '111:22:33.4': timedelta(hours = 111, minutes = 22, seconds = 33, milliseconds = 400),
         '111:22:33.04': timedelta(hours = 111, minutes = 22, seconds = 33, milliseconds = 40),
         '111:22:33.40': timedelta(hours = 111, minutes = 22, seconds = 33, milliseconds = 400),
+        '111:22:33.400': timedelta(hours = 111, minutes = 22, seconds = 33, milliseconds = 400),
+        '111:22:33.040': timedelta(hours = 111, minutes = 22, seconds = 33, milliseconds = 40),
+        '111:22:33.004': timedelta(hours = 111, minutes = 22, seconds = 33, milliseconds = 4),
+        '111:22:33.0045': timedelta(hours = 111, minutes = 22, seconds = 33, milliseconds = 4.5),
     }
 
     _type_class = ebuttdt.FullClockTimingType
@@ -61,6 +78,10 @@ class TestLimitedClockTimingType(TestCase):
         '11:22:33.4': timedelta(hours = 11, minutes = 22, seconds = 33, milliseconds = 400),
         '11:22:33.04': timedelta(hours = 11, minutes = 22, seconds = 33, milliseconds = 40),
         '11:22:33.40': timedelta(hours = 11, minutes = 22, seconds = 33, milliseconds = 400),
+        '11:22:33.400': timedelta(hours = 11, minutes = 22, seconds = 33, milliseconds = 400),
+        '11:22:33.040': timedelta(hours = 11, minutes = 22, seconds = 33, milliseconds = 40),
+        '11:22:33.004': timedelta(hours = 11, minutes = 22, seconds = 33, milliseconds = 4),
+        '11:22:33.0045': timedelta(hours = 11, minutes = 22, seconds = 33, milliseconds = 4.5),
     }
 
     _type_class = ebuttdt.LimitedClockTimingType

setup.py

@@ -31,8 +31,9 @@
         "ipdb",  # This will eventually be removed from here
         "configobj",
         "pyyaml",
+        "service_identity",
         "twisted",
-        "autobahn",
+        "autobahn<18",
         "nltk",
         "sortedcontainers",
         "configman",

testing/bdd/features/validation/timeBase_timeformat_constraints.feature

@@ -219,4 +219,4 @@ Scenario: Times in documentStartOfProgramme do cause validation error
   | xml_file                        | time_base  | start_time   |
   | timeBase_timeformat.xml         | media      | 00:00:00:00  |
   | timeBase_timeformat.xml         | clock      | 00:00:60     |
-  | timeBase_timeformat.xml         | smpte      | 10:00:00.00  |
+  | timeBase_timeformat.xml         | smpte      | 10:00:00.00  |