randomize on a directory occasionally duplicates since 2.0

[fourofspades]

Previously, an entire directory of files was played, in random order, without duplicates. It feels there is a regression in this area, as I am seeing occasional duplicate files being played in close order to each other.

My directory has about 2200 songs, and in my log file, even after 300 or so tracks have been streamed, I am seeing several duplicates.

My script:

#!/usr/bin/liquidsoap -v

#CONFIG
settings.log.stdout.set(true)
settings.init.allow_root.set(true)
settings.scheduler.fast_queues.set(2)
#settings.log.level.set(4)
#settings.server.log.level.set(4)
settings.decoder.file_extensions.flac.set(["flac"])
settings.decoder.file_extensions.mp3.set(["mp3"])
settings.decoder.file_extensions.mp4.set(["m4a", "m4b", "m4p", "m4v", "m4r", "3gp", "mp4"])

def log_song(m)
artist = m["artist"]
title = m["title"]
timestamp = time()
file.write(append=true, data="#{timestamp}      #{artist} - #{title}\n", "/music/history")
end

#NEW SOURCE
myradio=mksafe(normalize(playlist(mode='randomize',reload=1,reload_mode="rounds", "/music/")))
myradio.on_track(log_song)


#OUTPUT HIGH BITRATE
output.icecast(%fdkaac(channels=2, samplerate=44100, bitrate=192, afterburner=false, aot="mpeg4_aac_lc", transmux="adts", sbr_mode=false),
host="127.0.0.1",
port=8000,
password="redacted",
description="My Stream Title (High Bitrate)",
url="http://myurl:8000/high",
public=false,
icy_metadata="true",
genre="Rock",
mount="high",
myradio )

#OUTPUT LOW BITRATE
output.icecast(%fdkaac(channels=2, samplerate=44100, bitrate=64, afterburner=false, aot="mpeg4_aac_lc", transmux="adts", sbr_mode=false),
host="127.0.0.1",
port=8000,
password="redacted",
description="My Stream Title (low Bitrate)",
url="http://myurl:8000/low",
public=false,
genre="Rock",
icy_metadata="true",
mount="low",
myradio )

[smimram]

Let's find out where it might come from... Do you see any duplicates when you run this?

liquidsoap 'print(string.concat(separator="\n", playlist.files("/music/")))'

[fourofspades]

So I have a log file of the history, showing repeats, and a log file of the directory contents. The only curious thing is a metadata directory that my NAS (synology) creates, with a subfolder as the sane name as the file @eadir\Myfilename.flac\some stuff.

so for example:

/music/@eaDir/Rod Stewart - Maggie May.flac/01APIC_03.jpg
/music/@eaDir/Rod Stewart - Maggie May.flac/SYNOAUDIO_01APIC_03.jpg
/music/@eaDir/Rod Stewart - Maggie May.flac/SYNOINDEX_MEDIA_INFO
/music/Rod Stewart - Maggie May.flac

I have the full directory and history file if you want to see more.

[fourofspades]

Directory contents: https://controlc.com/782db975
History: https://controlc.com/150ee10d

PW: your project name, all lower-case

Example duplicate:

2021/11/24 03:09:36 Brody Dalle - Rat Race
2021/11/24 07:51:24 Brody Dalle - Rat Race

There are loads of others.

[smimram]

Thanks this indeed seems to be the culprit, your ls contains 1823 files and the output of playlist.files has 9855 entries. I'll investigate this further.

[smimram]

Ok so I am testing with the following script

#!/usr/bin/env liquidsoap
dir = argv(1)
print("Directory #{dir}")
ls = process.read.lines("find #{dir}")
print("ls: #{list.length(ls)-1} files")
pl = file.ls(absolute=true, recursive=true, dir)
print("pl: #{list.length(pl)} files")

Do you seen a difference in the numbers when you run the following?

script.liq /music/

I tried on several directories here and always got the same number.

By any chance, do you happen to have symbolic links in your /music/ directory?

[fourofspades]

Numbers are the same:

Directory /music/
ls: 6942 files
pl: 6942 files

[spy-reality]

I noticed that too. Also noticed that random repeats songs at the same time after reloading. Something is wrong.

[fourofspades]

Seeing same behaviour in 2.01 Is there any way I can exclude the parsing of the @eadir directory to see if that helps things?

@smimram have you been able to reproduce this issue? Do you need any more information from me?

[smimram]

I can now reproduce with 10 files with this example with 10 files it is already quite bad:

Song: files/1000.mp3 (0.%)
Song: files/1003.mp3 (10.%)
Song: files/1005.mp3 (20.%)
Song: files/1007.mp3 (30.%)
Song: files/1006.mp3 (40.%)
Song: files/1001.mp3 (50.%)
Song: files/1002.mp3 (60.%)
Song: files/1004.mp3 (70.%)
Song: files/song.mp3 (80.%)
Song: files/1008.mp3 (90.%)
Reloading.
Song: files/1007.mp3 (0.%)
Song: files/1005.mp3 (10.%)
Song: files/song.mp3 (20.%)
Song: files/1006.mp3 (30.%)
Song: files/1002.mp3 (40.%)
Song: files/1003.mp3 (50.%)
Song: files/1007.mp3 (60.%)
It seems that we have already played this file!...
Song: files/1004.mp3 (70.%)
Song: files/1000.mp3 (80.%)
Song: files/1001.mp3 (90.%)
Song: files/1008.mp3 (100.%)
Reloading.
Song: files/1008.mp3 (0.%)
Song: files/1006.mp3 (10.%)
Song: files/1003.mp3 (20.%)
Song: files/1007.mp3 (30.%)
Song: files/1000.mp3 (40.%)
Song: files/1001.mp3 (50.%)
Song: files/1005.mp3 (60.%)
Song: files/1008.mp3 (70.%)
It seems that we have already played this file!...
Reloading.
Reloading.
Song: files/song.mp3 (0.%)
Song: files/1003.mp3 (10.%)
Song: files/1007.mp3 (20.%)
Song: files/1004.mp3 (30.%)
Song: files/1000.mp3 (40.%)
Song: files/1006.mp3 (50.%)
Song: files/1000.mp3 (60.%)
It seems that we have already played this file!...
Song: files/1003.mp3 (70.%)
It seems that we have already played this file!...
Song: files/1008.mp3 (80.%)
Song: files/1007.mp3 (90.%)
It seems that we have already played this file!...
Song: files/1000.mp3 (100.%)
It seems that we have already played this file!...

stay tuned...

[smimram]

Ok, so I have just pushed a few fixes which handle bad corner-cases of playlists. I cannot anymore reproduce playlist duplicates with my above script. Can you confirm that this is ok for you too now?

[fourofspades]

Running build from docker main: Liquidsoap 2.1.0+git@e275bb8

Will report back in the morning, once it's done a few hundred or so tracks.

[smimram]

So, how is it going?

[fourofspades]

270 tracks in, and no duplicates. It's looking like it's resolved. I would have seen quite a few by now.

[smimram]

Yes! :)

Closing for now, feel free to reopen if you encounter some duplicates.

[fourofspades]

Bad news, it had quite a few duplicates since I last checked. (17 duplicates)
history: https://controlc.com/0fa6d542 (password: projectname)
directory contents (unchanged since previous post)
regex tester https://regex101.com/r/C1CEtB/1

[toots]

I'm testing Sam's script on v2.0.2-preview with 270 tracks and haven't seen any issue yet.

[fourofspades]

It feels like the changes made so far have improved things, but there are still issues. My logs suggest not as many duplicates as before,. Bit it's still not going through all the available track before repeating.

…

On Sat, 25 Dec 2021, 19:11 Romain Beauxis, ***@***.***> wrote: I'm testing Sam's script on v2.0.2-preview with 270 tracks and haven't seen any issue yet. — Reply to this email directly, view it on GitHub <#2057 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOVZNJFA3YIJTVQ42E7N5CLUSYJMVANCNFSM5H76XD4Q> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[fourofspades]

Can I assume the fixes made so far for this issue have made it into 2.0.2 release? If so I will switch to that release and report back (I am currently on Liquidsoap 2.1.0+git@e275bb8 but would prefer to stay "on-piste").

Gut feeling is the initial fixes indeed improve the problem, I didn't see any issues initially, but on very large playlists I did eventually see issues.

If 2.0.2 has all the fixes so far, I will clear my logs and repeat the tests. If there is any additional logging to enable, please let me know.

[fourofspades]

OK, so running 2.0.2 for a few days, and I can already see some tracks duplicating already. Thinking about this some more, i could be possible that I am getting file loading issues, so I don't actually have the sized pool of songs I believe I have. Unfortunately, I can't easily spot that due to this @eadir metadata folder that also being loaded (see early post in the thread), my logs contain many file load errors, and I would be hunting for a needle in a haystack to spot genuine errors, not bogus errors.

Is there a way to filter out this @eadir subfolder from my playlist script above? this would allow me to spot file load errors easier.

[fourofspades]

Trying to prevent access via Linux directory permissions is a no-go, Liquidsoap just bails out.

Error -1: Exception raised: (Sys_error "/music/@eadir: Permission denied")
Raised by primitive operation at Builtins_files.(fun).readdir in file "lang/builtins_files.ml", line 341, characters 22-39

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[smimram]

Ok, I think I found the issue. We were using list.shuffle which was implemented as

list.sort(fun(_,_) -> random.bool()?-1:1, l)

which is using OCaml's List.sort, which is assuming a total order as argument, which is not all the case with un(_,_) -> random.bool()?-1:1. I am pushing a fix to use good old Fischer-Yates instead...

[fourofspades]

Keen to try this out. I use docker, how would be the best way to get this? (I'm still mighty confused by branching model used).

[toots]

On docker, we push to the savonet/liquidsoap repository. The tags for the rolling release with the fix is: rolling-release-v2.1.x. The latest build just finished so if you pull this image, you should have the fix included in it.