LV: Clarify input plugin data upload semantics and define requirements

[kaixiong]

The upload() implementation of the various input plugins have subtly different and inconsistent characteristics that can cause synchronisation problems.

The current rendering logic (in LV::Bin) behaves best when:

1. upload() returns a copy of the previous uploaded data if input cannot keep up with call frequency
2. upload() drops samples that were not read in time

Please note that the above are not recommended as requirements. I discuss my reasons after the next section.

Below is an overview of how individual input plugins actually behave:

• input_pulseaudio violates (1) and (2), causing stuttering when call rate is too high, and lag when call rate is too low.
• input_debug violates (1) and (2) but has internal logic to slow the data generation down.
• input_mplayer observes (1) and (2), as it simply copies a fixed intermediary buffer written to by an external mplayer process during playback.
• input_wavein observes (1) and (2) by virtue of its threaded implementation and behaves like input_mplayer
• input_alsa observes the effects of (1) and (2) by draining all input data and passing it into LV::Audio
• input_jack observes (1) and (2) works in a similar way as input_wavein, but without explicit threading as JACK asynchronously calls the plugin-supplied handler to process new input data. It also uses a single buffer which may cause weird artifacts when it is both read and written at the same time,

The biggest challenge for the input system is to handle situations where the upload() call rate is either too low or too high. Both can happen due to the variability of frame rendering times that depend very much on specific actors and rendering dimensions.

(1) and (2) shouldn't be defined as requirements. The biggest reason is that duplicated or dropped samples can screw up beat detection and spectral analysis. It is best to keep the input as faithful as possible (up to a certain length of time to conserve memory).

[hartwork]

@kaixiong I am aware of these three buckets regarding input API's behavior:

• a) synchronous blocking (e.g. PulseAudio's Simple API)
• b) synchronous non-blocking (e.g. ALSA)
• c) asynchronous (i.e. through callbacks) (e.g. PulseAudio's Asynchronous API)

I believe we need to support all of these some way to not exclude some input API by design, and I conclude that:

• (a) cannot be in the same thread that is calling actor rendering, so we need a dedicated thread,
• (b) could be in the same thread as long as it's called often enough to keep latency down.
• (c) already has a dedicated thread, so only synchronation to do.

To summarize, my current impression is that libvisual needs to move input upload calls to a dedicated thread and out of the render thread. Happy to discuss more here or elsewhere.

[kaixiong]

@hartwork Nicely summarized. I agree with having a separate input upload thread. It would be a nice and relatively simple introduction of threading into libvisual that dovetails with long-term plans of using multiple cores for rendering.

[hartwork]

@kaixiong PS: For case (c) with asynchronous callbacks it would be great to have permanent (or global) access to the current VisAudio instance (or audio->samplepool) to be able to push audio with visual_audio_samplepool_input right from the callback.

[kaixiong]

@hartwork, can you explain why having a permanent/global VisAudio instance would be great? I default to skepticism about shared mutable objects (which languages like Rust ban in safe code for good reasons).

[hartwork]

@kaixiong the problem I am trying to solve is this:

The .upload function of the input plugin receives VisAudio *audio which allows pushing audio to audio->samplepool. Let's say we're implementing a plugin for an asynchronous audio API that serves data through us to a callback, and now the question arises: how does the callback get access to VisAudio *audio or at least the sample pool? One solution is to not have thr callback access it directly but to write data into some other buffer and leave direct access to audio->samplepool: the callback writes the buffer, .upload reads the buffer and puts it into the sample pool. That means we need that additional buffer space and access to it either needs locking or complex lock-free approaches.

If instead .upload would store the pointer to audio->samplepool in say priv->audio_samplepool, the callback could access it but then it's (in theory) unclear if the samplepool is still current, has not been freed, if passing it on was "allowed" in a momory ownership sense. I guess then the samplepool would need to be locked on access but it doesn't yet support locking at all? 🤔

What do you think?

[kaixiong]

@hartwork Ah I see what you mean. It is a classic producer-consumer problem. I don't have concrete ideas yet but here are quick thoughts:

1. Each input plugin should only be concerned with producing audio samples. The framework takes care of the consuming end.
2. Each input plugin should have its own data queue, an entity that's separate from VisAudio. So if there is something persistent, it would be this.
3. VisAudio should be kept simple, nothing more than a 'dumb' collection of audio channel samples. That is, a value object.
4. The produced audio samples should have timestamps, perhaps based on a pipeline-wide clock that counts from zero the moment the pipeline runs. This caters for interruptions or deliberate pauses. In such scenarios, the input plugin does not need to generate fillers.
5. On the consumption end, there would be a ring buffer managed by the framework for each input that holds up to n (milli)seconds of the most recent samples. The rest of the pipeline e.g. actors read from this buffer.