following to Feature Request: Implementing Persistent Speaker Embeddings Across Conversations #227

[DmitriyG228]

Hey @juanmc2005,

Following to the issue #227,

• I have implemented speaker centroids as list of embeddings without mapping, assuming speaker_0 == centroid 0 inside SpeakerDiarization
• added a better version of RedisWriter
• It looks like importing redis is unnessessary if we are passing a redis_client object into RedisWriter

Thank you!

[DmitriyG228]

that's how I run this

from diart import SpeakerDiarization
from diart.sources import FileAudioSource
from diart.inference import StreamingInference
from diart.sinks import  RedisWriter

pipeline = SpeakerDiarization(return_embeddings=True) # return_embeddings

audio_file_path = path  # Replace with the path to your audio file
sample_rate = 16000
file_source = FileAudioSource(audio_file_path,sample_rate)  # Use FileAudioSource

inference = StreamingInference(pipeline, file_source, do_plot=False)
inference.attach_observers(RedisWriter(file_source.uri, redis_client)) # RedisWriter instead of RTTMWriter

prediction = inference()

[juanmc2005]

Thanks for this PR! I have several questions about the best way to implement the centroid retrieval. Centroid setting for the beginning of a conversation is missing in the current code.

Also, please make sure to run the black formatter before pushing

[juanmc2005]

Why do we need to return centroids with each output? In order to have persistent centroids it would be enough to have a way of extracting them after inference and a way of setting them before inference. I think the best way to do this would be to expose getters and setters.

The thing is I don't want the RedisWriter to be exclusively for diarization. I would like for it to be compatible with VAD and in the future with transcription too.

Adding things at the end of each output is not a sustainable solution. With time, we could end up having huge tuples. An alternative would be to have different output objects per pipeline, like DiarizationOutput and VADOutput, but I want all sinks to be able to handle outputs seamlessly so they should remain at least similar enough. This would also require big changes to all inference and sink objects.

[DmitriyG228]

I think, we need to have setnroings ducing inference in the real time application. Obviously, every iteration is overkill

it would be enough to have a way of extracting them after inference

My usecase is about global speakers identification in real - time environment, which means I want to obtain speaker embeddings at some points for vector store lookup. After inference is not enough. As for now, I think good idea is to obtain a centroid for each new speaker after it's well - converged. Getting embeddings at each iteration obviously is overkill, but that worka for me at this point, as I can keep only the last state of centroids downstream. Maybe, a better idea is to develop an on-demand method for that, but I am pretty new to real time application and not sure how to do it right. Exposing getters looks good to me, but I am not sure how to use them

[juanmc2005]

Use _extract_prediction() here to get the annotation object instead of using an if statement

[juanmc2005]

This attribute is not needed. self.uri is enough

[juanmc2005]

Let's not name this "diarization". We could also be writing the output of a VAD pipeline here

[juanmc2005]

patch_collar isn't used here, right? In that case it should be removed

[juanmc2005]

Remove blank comment

[juanmc2005]

Remove blank comment

[DmitriyG228]

Centroid setting for the beginning of a conversation is missing in the current code.

Do you have specific use case for centroids setting, do you find it helpful?

[juanmc2005]

@DmitriyG228 if we can't set speaker centroids in the beginning, then this is not implementing "persistent speaker embeddings across conversations". Maybe I'm not understanding the use case very well, could you clarify this?

[vtontodonato]

I have a similar requirement, I need to recognize the "moderator" in a conversation, I am sure the moderator speaks for the first 30 seconds, I do not need the recognizing of the other speakers, but I need to be sure when the moderator is speaking again. A sort of "fix this centroid forever", the others I do not care.

There is any way to do such operation in the current implementation? There is the possibility to store the "embeddng" of a voice and just recognize such speaker during a conversation? The others speakers can be wrong I'll not use that.

[juanmc2005]

@vtontodonato that would be an interesting feature to add, but I think it's unrelated to this PR. If you're up for it, I would suggest adding a freeze_centroids(centroids: list[int]) method to OnlineSpeakerClustering, where you would simply keep track of the "frozen" centroids and prevent their updates in identify(). I would gladly merge a PR with this feature!

[vtontodonato]

I'll give a look at such methods ... and send you modification I'll be able to do to fit DIART in my scenario.