Repository navigation
Chunked encoding
It is possible to do parallel chunked encoding with encore, where the input is conceptually split into 'chunks' which are transcoded in parallel. Chunked encoding is enabled for a job by setting the segmentLength property to a non-zero value.
For details on how to submit jobs with chunked encoding, see the API and Job Submission guide.
When chunked encoding is enabled for a job, the job will be split into independent 'tasks' where each task is responsible for encoding a chunk of video and/or audio, or possibly complete audio depending on the selected audio encoding mode (see below). Each task will then be put on the encore job queue until it gets picked up by an encore instance.
To be able to use chunked encoding, the encore cluster must be configured with shared storage that is mounted at the same point in all instances. The path to this directory should be set with the
encore-settings.sharedWorkDir property, or alternatively the ENCORESETTINGS_SHAREDWORKDIR env variable.
It is also a good idea to configure a small poll interval, ie ENCORE_SETTINGS_POLL_DELAY=100ms .
The segmentLength property controls how long each chunk is. The value selected for segmentLength MUST be a multiple of the output GOP-length of the output to get frame-accurate output. Also note that if the input is not in an all-intra format, the output may possibly not be frame accurate unless the segmentLength is selected in such a way that each chunk starts on an IDR-frame in the input.
For maximum performance, segmentLength should be selected such that the number of chunks is at least equal to the maximum number of parallel jobs for the encore cluster. Increasing the number of chunks above that does usually not increase performance. Setting segmentLength to a 'too small' value may have a negative performance impact due to the small overhead associated with processing of each chunk.
When using chunked encoding, there are three different audio encoding modes.
- ENCODE_WITH_VIDEO - Audio is encoded together with each video segment.
- ENCODE_SEPARATELY_FULL - Audio is encoded without chunking and then muxed with video segments as necessary
- ENCODE_SEPARATELY_SEGMENTED - Audio is split into chunks and encoded in parallel with video
Which mode to use can be selected by setting the audioEncodingMode on the job. If this property is not set on the job, the audio encoding mode will be selected based on the encore-settings.encoding.segmented-encoding.audio-encoding-mode configuration property. If none of these properties are set, ENCODE_WITH_VIDEO will be used.
For backwards compatibility, this is the default mode when no other mode is specified. In this mode audio encoding is chunked in the same way as video encoding. This mode is only recommended when all audio track are muxed with video tracks, ie all audio encodes of the encoding profiles are contained within video encodes. Using this mode with audio only can be expected to result in glitches in the output.
In this mode, a separate task is created for audio encoding. This task will encode all audiostreams output. The audio tracks will get muxed with the joined video tracks when necessary, ie for output with muxed audio and video.
In this mode, audio is encoded separately from video in chunks. The duration of each audio chunk can be specified with the optional audioSegmentLength property of the job. The actual encoded chunk will have some padding added at the start and end. This padding is removed when the segments are joined together.
Note that the audioSegmentLength, if set, should be set to a multiple of the duration of an audio frame. If this property is not set, encore will select an appropriate value based on the audio sample rate. For the common sample rates 48kHz and 44.1kHz a 256s audio segment length will be used.
Note that this audio encoding mode can only be used together with the aac/libfdk_aac codecs.
If you are certain that it works for your usecase, use ENCODE_WITH_VIDEO. In most other cases it is best to use ENCODE_SEPARATELY_FULL. If audio encoding speed is a bottleneck, ENCODE_SEPARATELY_SEGMENTED may be used to increase encoding speed.