Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add hls support #2794

Open
wants to merge 27 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
7efe718
hls support first commit
mp3butcher Dec 1, 2023
08b9d0e
Update config.yml for hls config
mp3butcher Dec 1, 2023
ed56081
remove useless fields
mp3butcher Dec 1, 2023
17f47ca
CI compliance
mp3butcher Dec 1, 2023
bc5050c
Update install and config
mp3butcher Dec 1, 2023
3119454
make compatible with liquidsoap 1.4 output.file.hls prototype
mp3butcher Dec 2, 2023
c62cb8f
liquidsoap 2.2 removed streaminfos parameter so crossversion hls patt…
mp3butcher Dec 12, 2023
8f3bde7
add an alias to playout hls output
mp3butcher Dec 12, 2023
61aa1fb
use SERVER_PORT for hls stream
mp3butcher Dec 12, 2023
76010bc
liquidsoap 1.4 require differents streams bitrates for player to select
mp3butcher Dec 12, 2023
45af9ed
remove useless host port settings
mp3butcher Dec 13, 2023
faa3f86
document hls output
mp3butcher Dec 13, 2023
7da16a1
add an alias to playout hls output
mp3butcher Dec 13, 2023
a439af8
simplification
mp3butcher Dec 13, 2023
4872cae
add HlsStream to imports
mp3butcher Dec 13, 2023
5ee1e84
Update configuration.md
mp3butcher Dec 13, 2023
28e83ad
add missing changes
mp3butcher Dec 17, 2023
2723f2b
add hls testing
mp3butcher Dec 18, 2023
2d62117
change hls configuration
mp3butcher Dec 18, 2023
0833239
make hls output path an install setting
mp3butcher Dec 20, 2023
44f2dae
update test
mp3butcher Jan 19, 2024
95316d6
update configuration files and manual
mp3butcher Jan 19, 2024
07d9448
setup docker files
mp3butcher Jan 19, 2024
3a6aefd
Merge remote-tracking branch 'upstream/main' into hls
mp3butcher Jan 19, 2024
6f76da0
merge misses
mp3butcher Jan 19, 2024
06cf701
test miss
mp3butcher Jan 19, 2024
c7e95fd
update pytest snapshot
mp3butcher Jan 19, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions docker/config.template.yml
Original file line number Diff line number Diff line change
Expand Up @@ -317,6 +317,53 @@ stream:
# > default is false
mobile: false

hls:
- # Whether the output is enabled.
# > default is false
enabled: true
# > must be one of ('mpegts', 'mp3', 'adts', 'mp4')
# > this field is REQUIRED
format: mpegts
# > segment_duration (default:2.0)
segment_duration: 2.0
# > segments count (default:5)
segment_count: 5
# > segments_overhead (default:5)
segments_overhead: 5
# Output public url, If not defined, the value will be generated from
# the [general.public_url] hostname, the output port and mount.
public_url:
# web server host.
# > default is localhost
host: localhost
# webs server server port.
# > default is 80
port: 80
# hls mount point
# > this field is REQUIRED
mount: hls/main
streams:
- # > prefix of generated fragment
# > this field is REQUIRED
fragment_prefix: mp3_low
# > must be one of ( 'libmp3lame', 'flac', 'aac', 'libopus', 'libvorbis')
# > this field is REQUIRED
codec: libmp3lame
# > bitrate of the stream
# > this field is REQUIRED
bitrate: 32k
# > sampling rate (default: 44100Hz)
sample_rate: 44100
- fragment_prefix: mp3_med
codec: libmp3lame
bitrate: 128k
- fragment_prefix: mp3_hifi
codec: libmp3lame
bitrate: 256k
# Whether the stream should be used for mobile devices.
# > default is false
mobile: false

# System outputs.
# > max items is 1
system:
Expand Down
47 changes: 47 additions & 0 deletions docker/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -317,6 +317,53 @@ stream:
# > default is false
mobile: false

hls:
- # Whether the output is enabled.
# > default is false
enabled: true
# > must be one of ('mpegts', 'mp3', 'adts', 'mp4')
# > this field is REQUIRED
format: mpegts
# > segment_duration (default:2.0)
segment_duration: 2.0
# > segments count (default:5)
segment_count: 5
# > segments_overhead (default:5)
segments_overhead: 5
# Output public url, If not defined, the value will be generated from
# the [general.public_url] hostname, the output port and mount.
public_url:
# web server host.
# > default is localhost
host: localhost
# webs server server port.
# > default is 80
port: 80
# hls mount point
# > this field is REQUIRED
mount: hls/main
streams:
- # > prefix of generated fragment
# > this field is REQUIRED
fragment_prefix: mp3_low
# > must be one of ( 'libmp3lame', 'flac', 'aac', 'libopus', 'libvorbis')
# > this field is REQUIRED
codec: libmp3lame
# > bitrate of the stream
# > this field is REQUIRED
bitrate: 32k
# > sampling rate (default: 44100Hz)
sample_rate: 44100
- fragment_prefix: mp3_med
codec: libmp3lame
bitrate: 128k
- fragment_prefix: mp3_hifi
codec: libmp3lame
bitrate: 256k
# Whether the stream should be used for mobile devices.
# > default is false
mobile: false

# System outputs.
# > max items is 1
system:
Expand Down
47 changes: 47 additions & 0 deletions docker/example/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -317,6 +317,53 @@ stream:
# > default is false
mobile: false

hls:
- # Whether the output is enabled.
# > default is false
enabled: true
# > must be one of ('mpegts', 'mp3', 'adts', 'mp4')
# > this field is REQUIRED
format: mpegts
# > segment_duration (default:2.0)
segment_duration: 2.0
# > segments count (default:5)
segment_count: 5
# > segments_overhead (default:5)
segments_overhead: 5
# Output public url, If not defined, the value will be generated from
# the [general.public_url] hostname, the output port and mount.
public_url:
# web server host.
# > default is localhost
host: localhost
# webs server server port.
# > default is 80
port: 80
# hls mount point
# > this field is REQUIRED
mount: hls/main
streams:
- # > prefix of generated fragment
# > this field is REQUIRED
fragment_prefix: mp3_low
# > must be one of ( 'libmp3lame', 'flac', 'aac', 'libopus', 'libvorbis')
# > this field is REQUIRED
codec: libmp3lame
# > bitrate of the stream
# > this field is REQUIRED
bitrate: 32k
# > sampling rate (default: 44100Hz)
sample_rate: 44100
- fragment_prefix: mp3_med
codec: libmp3lame
bitrate: 128k
- fragment_prefix: mp3_hifi
codec: libmp3lame
bitrate: 256k
# Whether the stream should be used for mobile devices.
# > default is false
mobile: false

# System outputs.
# > max items is 1
system:
Expand Down
1 change: 1 addition & 0 deletions install
Original file line number Diff line number Diff line change
Expand Up @@ -628,6 +628,7 @@ link_python_app libretime-playout-notify

info "creating libretime-playout working directory"
mkdir_and_chown "$LIBRETIME_USER" "$WORKING_DIR/playout"
mkdir_and_chown "$LIBRETIME_USER" "$WORKING_DIR/playout/hls"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer to have playout create the "$WORKING_DIR/playout/hls" directory, the "$WORKING_DIR/playout" is writable by libretime.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the hls directory must be created before nginx starts so we can't let playout create this directory...no?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why ? We only read in this directory during a request, and response with 404 if no files are found. Sound doable to me.

Copy link
Contributor Author

@mp3butcher mp3butcher Jan 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you think it's safe to launch nginx and serve a not-yet-created directory, so i'm good with removing this line but it will require testing (with docker it's required -liquidsoap can't create output directory-)


install_service "libretime-liquidsoap.service" "$SCRIPT_DIR/playout/install/systemd/libretime-liquidsoap.service"
install_service "libretime-playout.service" "$SCRIPT_DIR/playout/install/systemd/libretime-playout.service"
Expand Down
47 changes: 47 additions & 0 deletions installer/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -317,6 +317,53 @@ stream:
# > default is false
mobile: false

hls:
- # Whether the output is enabled.
# > default is false
enabled: true
# > must be one of ('mpegts', 'mp3', 'adts', 'mp4')
# > this field is REQUIRED
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's reduce the amount of format for now, extra format can be added and tested in future PRs.

Copy link
Contributor Author

@mp3butcher mp3butcher Dec 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AC3/ADTS is in the spec, further it work as it should with hls.js so i'll let it .

I'll remove mp4 because i haven't achieve anything with liquidsoap 1.4 even if in spec...:/

ok to remove mp3

Also, I'll will move format property as stream property to fit liquidsoap stream model (it was a mistake to put it as hls property)

edit mp4 seems to require movflags=+dash+skip_sidx+skip_trailer+frag_custom+global_header

format: mpegts
# > segment_duration (default:2.0)
segment_duration: 2.0
# > segments count (default:5)
segment_count: 5
# > segments_overhead (default:5)
segments_overhead: 5
# Output public url, If not defined, the value will be generated from
# the [general.public_url] hostname, the output port and mount.
public_url:
# web server host.
# > default is localhost
host: localhost
# webs server server port.
# > default is 80
port: 80
# hls mount point
# > this field is REQUIRED
mount: hls/main
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The word "mount" is from the icecast/shoutcast vocabulary, I am not sure we want to have if in the context of HLS.

I think this should also include the playlist extension .m3u8

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok for m3u8 but i don't know if we need to change the mount property name it's quite coherent for a user perspective...
What would you suggest? manifest?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you okay with these settings?:

stream:
  outputs:
    # HLS output manifests.
    # > max items is 10
    hls:
      - # Whether the output is enabled.
        # > default is false
        enabled: false
        # Output public url. If not defined, the value will be generated from
        # the [general.public_url] hostname/port.
        public_url:
        # hls playlist name.
        # > this field is REQUIRED
        playlist: "main.m3u8"
        # > segment_duration (default:2.0)
        segment_duration: 2.0
        # > segments count: segments count buffered (default:5)
        segment_count: 5
        # > segments_overhead: segments count kept past the playlist size for those listeners who are still listening on outdated segments. (default:5)
        segments_overhead: 5
        streams:
          - # > prefix of generated segments
            # > this field is REQUIRED
            segments_prefix: mp3_low
            # > format of the container must be one of ('mpegts', 'adts')
            # > this field is REQUIRED
            format: mpegts
            # > codec used for the stream, must be one of ( 'libmp3lame', 'aac')
            # > this field is REQUIRED
            codec: libmp3lame
            # > bitrate of the stream
            # > this field is REQUIRED
            bitrate: 32k
            # > audio sampling rate
            # > this field is REQUIRED
            sample_rate: 44100
          - fragment_prefix: mp3_hifi
            codec: libmp3lame
            bitrate: 256k
            sample_rate: 44100

streams:
- # > prefix of generated fragment
# > this field is REQUIRED
fragment_prefix: mp3_low
# > must be one of ( 'libmp3lame', 'flac', 'aac', 'libopus', 'libvorbis')
# > this field is REQUIRED
codec: libmp3lame
# > bitrate of the stream
# > this field is REQUIRED
bitrate: 32k
# > sampling rate (default: 44100Hz)
sample_rate: 44100
- fragment_prefix: mp3_med
codec: libmp3lame
bitrate: 128k
- fragment_prefix: mp3_hifi
mp3butcher marked this conversation as resolved.
Show resolved Hide resolved
codec: libmp3lame
bitrate: 256k
# Whether the stream should be used for mobile devices.
# > default is false
mobile: false

# System outputs.
# > max items is 1
system:
Expand Down
4 changes: 4 additions & 0 deletions installer/nginx/libretime.conf
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,10 @@ server {
client_max_body_size 512M;
client_body_timeout 300s;

location /hls {
jooola marked this conversation as resolved.
Show resolved Hide resolved
alias /var/lib/libretime/playout/hls;
mp3butcher marked this conversation as resolved.
Show resolved Hide resolved
mp3butcher marked this conversation as resolved.
Show resolved Hide resolved
}

location ~ \.php$ {
fastcgi_buffers 64 4K;
fastcgi_split_path_info ^(.+\.php)(/.+)$;
Expand Down
34 changes: 33 additions & 1 deletion legacy/application/configs/conf.php
Original file line number Diff line number Diff line change
Expand Up @@ -204,6 +204,37 @@ public function getConfigTreeBuilder()
/* */->booleanNode('mobile')->defaultFalse()->end()
/**/->end()->end()->end()

// Hls outputs
/**/->arrayNode('hls')->arrayPrototype()->children()
/* */->arrayNode('streams')->arrayPrototype()->children()
/* */->scalarNode('fragment_prefix')->end()
/* */->scalarNode('bitrate')->isRequired()->end()
/* */->IntegerNode('sample_rate')->defaultValue(44100)->end()
/* */->scalarNode('codec')->cannotBeEmpty()
/* */->validate()->ifNotInArray(['aac', 'libmp3lame', 'flac', 'libopus', 'libvorbis'])
/* */->thenInvalid('invalid stream.outputs.hls.streams.codec %s')
/* */->end()
/* */->end()
/* */->end()->end()->end()
/* */->scalarNode('format')->cannotBeEmpty()
/* */->validate()->ifNotInArray(['mpegts', 'mp3', 'adts', 'mp4'])
/* */->thenInvalid('invalid stream.outputs.hls.format %s')
/* */->end()
/* */->end()
/* */->floatNode('segment_duration')->defaultValue(2.0)->end()
/* */->integerNode('segment_count')->defaultValue(5)->end()
/* */->integerNode('segments_overhead')->defaultValue(5)->end()
/* */->booleanNode('enabled')->defaultFalse()->end()

/* */->enumNode('kind')->values(['hls'])->defaultValue('hls')->end()
/* */->scalarNode('public_url')->end()
/* */->scalarNode('host')->defaultValue('localhost')->end()
/* */->integerNode('port')->defaultValue(80)->end()
/* */->scalarNode('mount')->cannotBeEmpty()
/* */->validate()->ifString()->then($trim_leading_slash)->end()
/* */->end()
/* */->booleanNode('mobile')->defaultFalse()->end()
/**/->end()->end()->end()
// System outputs
/**/->arrayNode('system')->arrayPrototype()->children()
/* */->booleanNode('enabled')->defaultFalse()->end()
Expand Down Expand Up @@ -270,7 +301,8 @@ private static function load()
// Merge Icecast and Shoutcast outputs
$values['stream']['outputs']['merged'] = array_merge(
$values['stream']['outputs']['icecast'],
$values['stream']['outputs']['shoutcast']
$values['stream']['outputs']['shoutcast'],
$values['stream']['outputs']['hls']
Comment on lines +301 to +302
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's not merge HLS streams in the icecast/shoutcast streams. We can benefit from a fresh/clean settings schema, and I don't see the benefit from merging this in the "merged" streams.

Copy link
Contributor Author

@mp3butcher mp3butcher Dec 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's mandatory to do that, it doesn't work if i remove this line (merged is the array parsed in StreamSettings.php)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it mandatory? "It doesn't work" doesn't seem like a valid argument sorry 😉

Copy link
Contributor Author

@mp3butcher mp3butcher Jan 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Streams settings data (url,codec..) presented in the player are retrieved from this merged outputs array

);

self::$values = $values;
Expand Down
14 changes: 9 additions & 5 deletions legacy/application/models/StreamSetting.php
Original file line number Diff line number Diff line change
Expand Up @@ -55,16 +55,20 @@ public static function getOutput($key, $add_prefix = false)
$prefix . 'pass' => $output['source_password'] ?? '',
$prefix . 'admin_user' => $output['admin_user'] ?? 'admin',
$prefix . 'admin_pass' => $output['admin_password'] ?? '',
$prefix . 'channels' => $output['audio']['channels'] ?? 'stereo',
$prefix . 'bitrate' => $output['audio']['bitrate'] ?? 128,
$prefix . 'type' => $output['audio']['format'],
$prefix . 'name' => $output['name'] ?? '',
$prefix . 'description' => $output['description'] ?? '',
$prefix . 'genre' => $output['genre'] ?? '',
$prefix . 'url' => $output['website'] ?? '',
$prefix . 'mobile' => $output['mobile'] ?? 'false',
// $prefix . 'liquidsoap_error' => 'waiting',
];
if (array_key_exists('audio', $output)) {
$result .= merge([
$prefix . 'channels' => $output['audio']['channels'] ?? 'stereo',
$prefix . 'bitrate' => $output['audio']['bitrate'] ?? 128,
$prefix . 'type' => $output['audio']['format'],
]);
}
}

if (!$result[$prefix . 'public_url']) {
Expand Down Expand Up @@ -108,8 +112,8 @@ public static function getEnabledStreamData()
$prefix = $id . '_';
$streams[$id] = [
'url' => $streamData[$prefix . 'public_url'],
'codec' => $streamData[$prefix . 'type'],
'bitrate' => $streamData[$prefix . 'bitrate'],
'codec' => $streamData[$prefix . 'type'] ?? 'hls',
'bitrate' => $streamData[$prefix . 'bitrate'] ?? '',
'mobile' => $streamData[$prefix . 'mobile'],
];
}
Expand Down
19 changes: 19 additions & 0 deletions playout/libretime_playout/liquidsoap/1.4/ls_lib.liq
Copy link
Contributor Author

@mp3butcher mp3butcher Dec 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make a function because output.file.hls varies accross liquidsoap versions

Original file line number Diff line number Diff line change
@@ -1,3 +1,22 @@
def start_hls(input_main_mount, streamsref, servpath, segment_duration, segments, segments_overhead, segment_named, output_source)
streams_infos = ref([])
#generate dummy stream infos does the trick for 1.4
list.iter(fun(item) ->
streams_infos := list.append([(fst(item), (0,'',''))], !streams_infos )
, streamsref())
argstream = !streamsref
arginfos = !streams_infos
output.file.hls(playlist=input_main_mount,
segment_duration=segment_duration,
segments=segments,
segments_overhead=segments_overhead,
streams_info=arginfos,
segment_name=segment_named,
servpath,
argstream,
output_source )
end

def gateway(args)
command = "timeout --signal=KILL 45 libretime-playout-notify #{args} &"
log(command)
Expand Down
34 changes: 34 additions & 0 deletions playout/libretime_playout/liquidsoap/templates/outputs.liq.j2
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,33 @@
{%- endif -%}
{%- endmacro -%}


{#-
Build an hls output the output configuration.
#}
{%- macro output_hls(output_id, output) -%}
# hls:{{ output_id }}

output_hls_{{ output_id }}_source = s

streams = ref([])

def segment_name(~position,~extname, stream_name) =
timestamp = int_of_float(time())
duration = {{output.segment_duration}}
"#{stream_name}_#{duration}_#{timestamp}_#{position}.#{extname}"
end

start_hls(input_main_mount, streams,
"/var/lib/libretime/playout/hls",
{{output.segment_duration}},
{{output.segment_count}},
{{output.segments_overhead}},
segment_name,
output_hls_{{ output_id }}_source)

{%- endmacro -%}

{#-
Build an icecast output the output configuration.
#}
Expand Down Expand Up @@ -130,6 +157,13 @@ output.{{ output.kind.value }}(id="{{ output.kind.value }}:{{ loop.index }}", s)
{% endif -%}
{% endfor -%}

{% for output in config.stream.outputs.hls -%}
{% if output.enabled -%}
{{ output_hls(loop.index, output) }}

{% endif -%}
{% endfor -%}

{% for output in config.stream.outputs.icecast -%}
{% if output.enabled -%}
{{ output_icecast(loop.index, output) }}
Expand Down
1 change: 1 addition & 0 deletions shared/libretime_shared/config/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
DatabaseConfig,
GeneralConfig,
HarborInput,
HlsOutput,
IcecastOutput,
RabbitMQConfig,
ShoutcastOutput,
Expand Down
Loading
Loading