Load balance

[carrzkiss]

Hello everyone.
I'm finally getting to the point I can start working of the liquidsoap part of the very large project.

Doing Microsoft windows servers. I deal with load balancing our web servers to make sure that traffic runs smoothly at all times.

I researched about load balancing icecast servers and could only find one resource and that is doing RR-DNS.
A lot of the forms I read, everyone talked about the icecast servers can handle 10,000 + connections at a time and only limited by its hardware and available bandwidth.
--------
Let me make a scenario, and could someone tell me the best way to handle this.

100's of radio stations
1,000s of listeners per station.

How would you handle all the traffic?

I have the server's to install multiple Linux instances with icecast/liquidsoap running.

I just need to know what needs to be done to handle potential traffic.

Thank you.
Wayne.

[vitoyucepi]

Hi @carrzkiss,
Disclaimer. I've never done this myself, so I can't warn you about the potential pitfalls.

1. Check this presentation from fosdem 2020.
   After watching it, you will have a basic knowledge of the technologies.
2. Do you have any preferences for using Icecast instead of HLS?
3. If you'd like to use Icecast anyway, then you should dig further into the load balancing of HTTP long-polling connections.
4. Do you plan to use a cloud load balancer?

[carrzkiss]

Wanted to check in on this. It is mainly #1 & #4 that I am concerned about.

I love the idea that I can have 1,000s of listeners on a single server instance. But without some kind of load balance when that single server becomes overloaded can be an issue.

And with HLS, I just cannot find enough resources on using it. I know there has to be something out there that.

Since vitoyucepi mentioned
HTTP long-polling connections.
I can check in on that.

[Russsgithub]

You can deliver hls straight from a folder with nginx. Nginx location conf would be something like

        location /hls_s3 {
          resolver 8.8.8.8;
          proxy_http_version     1.1;
          proxy_redirect off;
          proxy_set_header       Connection "";
          proxy_set_header       Authorization '';
          proxy_set_header       X-Real-IP $remote_addr;
          proxy_set_header       X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_hide_header      x-amz-id-2;
          proxy_hide_header      x-amz-request-id;
          proxy_hide_header      x-amz-meta-server-side-encryption;
          proxy_hide_header      x-amz-server-side-encryption;
          proxy_hide_header      Set-Cookie;
          proxy_ignore_headers   Set-Cookie;
          proxy_intercept_errors on;
          add_header             Cache-Control no-cache;
          proxy_buffering        off;
          proxy_pass            # s3 url without trailing slash
        }

        location /hls {
          add_header 'Cache-Control' 'no-cache';
          alias /tmp/hls; # location of hls folder
          expires -1;

          access_log /var/log/nginx/hls.access.log; # I like to put the hls log in a different file
        }
        

The top one should work for s3 deliver the bottom for local.

This is the hls output code from the docs.

aac_lofi = %ffmpeg(format="mpegts",
                   %audio(
                     codec="aac",
                     channels=2,
                     ar=44100
                   ))

aac_midfi = %ffmpeg(format="mpegts",
                    %audio(
                      codec="aac",
                      channels=2,
                      ar=44100,
                      b="96k"
                    ))

aac_hifi = %ffmpeg(format="mpegts",
                   %audio(
                     codec="aac",
                     channels=2,
                     ar=44100,
                     b="192k"
                   ))

streams = [("aac_lofi",aac_lofi),
           ("aac_midfi", aac_midfi),
           ("aac_hifi", aac_hifi)]

def segment_name(~position,~extname,stream_name) =
  timestamp = int_of_float(gettimeofday())
  duration = 2
  "#{stream_name}_#{duration}_#{timestamp}_#{position}.#{extname}"
end

output.file.hls(playlist="live.m3u8",
                segment_duration=2.0,
                segments=5,
                segments_overhead=5,
                segment_name=segment_name,
                persist_at="/tmp/state.config",
                "/path/to/hls/directory",
                streams,
                source)

If you want to use s3 change the output.file.hls part to add on_file_change and sync you hls output folder to your s3 bucket, which will delete old files and upload the new ones including a relavant .m3u8 file which tell the player what to play.

output.file.hls(playlist="live.m3u8",
                segment_duration=2.0,
                segments=5,
                segments_overhead=5,
                segment_name=segment_name,
                persist_at="/tmp/state.config",
                "/tmp/hls",
                on_file_change = fun() -> process.run("aws s3 sync /tmp/hls/ <S3Uri>"), ### Upload to s3 ### This is off the top of my head so may need looking into or may be better to run in another thread.
                streams,
                source)

I use the local delivery method, but never have a huge amount to listeners. For 1000's of listeners CDN delivery (s3 / cloudfront etc) is probably the way to go. ... I have run a test of s3 delivery which worked , but have not put much time or testing into it.

......
You can you js libraries like hls.js or shaka player to play the stream on a webpage which have lots of stream management setting to look through and adjust to your needs.

I hope this helps a little.

Russ

[carrzkiss]

Hey, Russ.
That is a lot to go through.
I had to dig up one of my LIQ files, as all the Linux servers are shut down for the time being while we develop the visitor interface of the website itself.

This is the URL example for the Server Side playlist.
music = playlist("https://www.domain.com/Panels/DJ/101.asp",mode="normal")
(Each DJ will have their own channel ID, 102,103,104, etc.)
This works great for simply streaming music in the 101.asp playlist.

The information you provided is helpful, but I have no idea where to begin.

I am assuming it will be like this.
1: Install Linux Distro (Ubuntu Server Core (Non-Desktop) in my case)
2: Install Nginx
3: Install Liquidsoap (Do I install Icecast as well?)

So what would have to be done after 3?
Would I set up the LIQ script and then the HLS to run the LIQ scripts?

A lot of information I know.
Thanks to everyone so far.
Wayne

[Russsgithub]

OK.

You can use the playlist and hls code in one script.

e.g.

music = playlist(<playlist location>)

aac_lofi = %ffmpeg(format="mpegts",
                   %audio(
                     codec="aac",
                     channels=2,
                     ar=44100
                   ))

aac_midfi = %ffmpeg(format="mpegts",
                    %audio(
                      codec="aac",
                      channels=2,
                      ar=44100,
                      b="96k"
                    ))

aac_hifi = %ffmpeg(format="mpegts",
                   %audio(
                     codec="aac",
                     channels=2,
                     ar=44100,
                     b="192k"
                   ))

streams = [("aac_lofi",aac_lofi),
           ("aac_midfi", aac_midfi),
           ("aac_hifi", aac_hifi)]

def segment_name(~position,~extname,stream_name) =
  timestamp = int_of_float(gettimeofday())
  duration = 2
  "#{stream_name}_#{duration}_#{timestamp}_#{position}.#{extname}"
end

output.file.hls(playlist="live.m3u8",
                segment_duration=2.0,
                segments=5,
                segments_overhead=5,
                segment_name=segment_name,
                persist_at="/tmp/state.config",
                "/tmp/hls",
                streams,
                music)
                

no need for icecast unless you are using it, as in it's not needed to stream using hls.

This will output an aac encoded stream to /tmp/hls

Add the hls location block to you nginx conf, and you should have a stream you can access at /hls/live.m3u8

[carrzkiss]

To make sure I understand this correctly.
The url to the playlist 101.asp will create a file
101.m3u8
And this file will get updated when a change is made without the need to restart the station.
Is this correct?

Also.
With the
music101 = playlist()
output.file.hls(playlist="101.m3u8"

If there are multiple stations, would I create this section multiple times for every station ID?
music101 = playlist()
output.file.hls(playlist="101.m3u8
...code
music102 = playlist()
output.file.hls(playlist="102.m3u8

Would this be correct?

[Russsgithub]

There seems to be a discussion on Aws stream delivery at the liquidsoap event today. I'm guessing this will have something relevant to your project in.re. Creating sources, look into dynamic sources, I think the documents and maybe the book go into this.Good luck with you project.On 30 May 2023 06:15, carrzkiss ***@***.***> wrote: To make sure I understand this correctly. The url to the playlist 101.asp will create a file 101.m3u8 And this file will get updated when a change is made without the need to restart the station. Is this correct? Also. With the music101 = playlist() output.file.hls(playlist="101.m3u8" If there are multiple stations, would I create this section multiple times for every station ID? music101 = playlist() output.file.hls(playlist="101.m3u8 ...code music102 = playlist() output.file.hls(playlist="102.m3u8 Would this be correct? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

[carrzkiss]

OK, No answer to rather my theory was correct or not.
I will assume that is where the "Dynamic Sources" come into play, correct?

You can embed multiple LIQ files in the liquidsoap config file, as I have already done that.
I will check in on the "Dynamic Sources" and see what I can come up with.

Thanks, Rus.
If you can think of anything else I may need to know when I install the new Linux test system, that would be great.

Wayne

[Russsgithub]

OK.

You can use the playlist and hls code in one script.

e.g.

music = playlist(<playlist location>)

aac_lofi = %ffmpeg(format="mpegts",
                   %audio(
                     codec="aac",
                     channels=2,
                     ar=44100
                   ))

aac_midfi = %ffmpeg(format="mpegts",
                    %audio(
                      codec="aac",
                      channels=2,
                      ar=44100,
                      b="96k"
                    ))

aac_hifi = %ffmpeg(format="mpegts",
                   %audio(
                     codec="aac",
                     channels=2,
                     ar=44100,
                     b="192k"
                   ))

streams = [("aac_lofi",aac_lofi),
           ("aac_midfi", aac_midfi),
           ("aac_hifi", aac_hifi)]

def segment_name(~position,~extname,stream_name) =
  timestamp = int_of_float(gettimeofday())
  duration = 2
  "#{stream_name}_#{duration}_#{timestamp}_#{position}.#{extname}"
end

output.file.hls(playlist="live.m3u8",
                segment_duration=2.0,
                segments=5,
                segments_overhead=5,
                segment_name=segment_name,
                persist_at="/tmp/state.config",
                "/tmp/hls",
                streams,
                music)
                

no need for icecast unless you are using it, as in it's not needed to stream using hls.

This will output an aac encoded stream to /tmp/hls

Add the hls location block to you nginx conf, and you should have a stream you can access at /hls/live.m3u8

If playlist(".... .asp") generates the audio you need ( I don't know what the asp file does ) , you may need request.dynamic ?, Then the above code would generate an HLS stream inside /tmp/HLS consisting of a manifest file live.m3u8 which contains a list of "live" file in 3 different qualities, and the related audio files. All the files constantly change holding the last 20 secs of the stream. And all are updated as the stream progresses, so yes ... There is no need to restart the station .

Dynamic sources would allow you to create a new source and output on demand via telnet / an http(s) request, so you can turn things on / off as you may need. e.g add a playlist and HLS output for a new client.

I am a liquidsoap user and just thought I'd try to point you in direction that would, hopefully, help you get closer the the results your aiming for. I'd also recommend looking through the liquidsoap book and documents , which are a very good place to start and very informative.

[carrzkiss]

The .asp page is just a Database generated playlist.
That is all.
An example of the list is this.

https://domain/Asylum/01082021184726649.mp3
https://domain/Asylum/01082021184727383.mp3
https://domain/Asylum/01082021184727946.mp3

I will work on doing a new Linux VM later on in the week or over the coming weekend.
I will post my results and go from there.

You wrote this

There seems to be a discussion on Aws stream delivery at the liquidsoap event today.
Is there a video or an article on this available?

[jrhoward]

keep in mind HLS is not a standard and consequently is not supported without plugins in some web based clients.

[carrzkiss]

I am just now learning about HLS, so.
What plugins for what clients?

If I can do what I need to do with HLS then dealing with plugins may not be to much of a big deal
Depending on how many.

[carrzkiss]

I just read about the plugins.
iOS and Android come with it.
And desktop Chrome has a plugin so does Firefox.

Glad you came in and informed me of this situation.

I am going to continue to look at doing what I need without HLS while still working with HLS for the time being.

[jrhoward]

yeh I stream both with nginx in front and cache the hls files using the standard nginx proxy cache. In an HTML audio tag you can add multiple stream sources and the browser choses the one it supports

[jrhoward]

also I install/run liquidsoap on it's own machine, icecast and nginx are on a different machines and liquidsoap can push to multiple streamers. You can hide Harbor for hls or what ever behind your nginx cache

[carrzkiss]

Sounds interesting.
Gave you provide a write up on how you have everything set up?

I have a little more work to finish up the frontend of our website then I will start working on the streaming audio radio for it.

What you have explained sounds like what I have set up with my web and database servers.

[jrhoward]

I have a drawio svg I can paste here, but to put some context around my design, it was very much is an experiment for me and one of my main goals was to keep it cheap ( but also clean !!).

Based on the cloud provider it was cheaper for me to split into a couple of low spec vms than to run a single beefed up machine. The bandwidth, cpu, io etc is fixed so I get no bill shock .

I hide everything behind nginx to reduce the attack surface area.

It's all in terraform so I can spin up , spin down machines as required.

Load balancing tcp ( ie. icecast streaming ) with sticky sessions is really hard. A quick google suggested by IP only which won't guarantee evenly distributed traffic, or that a user retains the same IP.

If you expect most of your traffic to come from web browsers then maybe HLS will work for you if you embed an HLS client in your web UI eg. https://github.com/video-dev/hls.js/

[jrhoward]

[jrhoward]

based on above the load to the client is predictable as server scales out

[carrzkiss]

Very cool.

The information and link posted by Vito.
https://archive.fosdem.org/2020/schedule/event/om_audio_streaming/
They show where they have.
2 million daily listeners
200,000 concurrent listeners daily.

That is what I am interested in learning more about.
How do they handle the traffic to the channels giving the same live broadcast to their listeners.

Right now I am doing everything in-house.
In the future, depending on how the site grows will depend on rather I go with using the cloud. But right now, I am happy with my server setups now

I can run as many VMS as needed to get the job done.

There is a lot that needs to be done and learned over the course of the next year.
I am hoping to have the radio to go live some in mid-2024.

[vitoyucepi]

Hi @carrzkiss,
You could check the report from the recent liquidshop.
Here's the link to youtube and the slides.

I think their team specialization is SRE.
Among the key skills are

• linux
• cloud providers
• delivery
• monitoring