Design / Implement Audio and Payload containers

[ejhumphrey]

Currently, JAMS objects are being used via the top-level sandbox to ferry data through deformation pipelines. This is a little clunky for a few reasons, some more obvious than others. For my part, a big one is transforming JAMS without audio / transforming audio without JAMS.

The important thing to note though is that the JAMS object is pretty powerful, which makes it super easy to do things with and to it. We can't say the same for the audio signal, and the JAMS object doesn't (and shouldn't) offer similar functionality for wrangling muda history, for example.

I'd be keen to encapsulate audio and annotation data as separate attributes of a Payload object (or what have you) that can pass through the deformer pipeline agnostically. Putting some smarts into the different containers will also make it easier to introduce other audio deformations later, like stereo / spatialization, and keep good records on applied deformations.

And, as another win (in my book at least), it could allow us to leverage different audio reading/writing backends, which can be justifiable in different scenarios.

thoughts?

[bmcfee]

Currently, JAMS objects are being used via the top-level sandbox to ferry data through deformation pipelines. This is a little clunky for a few reasons, some more obvious than others. For my part, a big one is transforming JAMS without audio / transforming audio without JAMS.

Transforming jams without audio is technically no problem, though I forget if it actually works. (I think it should.)

Transforming audio without JAMS is also possible, in that you can have the jams pipe construct a dummy jams object.

I'd be keen to encapsulate audio and annotation data as separate attributes of a Payload object (or what have you) that can pass through the deformer pipeline agnostically. Putting some smarts into the different containers will also make it easier to introduce other audio deformations later, like stereo / spatialization, and keep good records on applied deformations.

This will need some thought, but I'm not totally opposed to it in principle.

And, as another win (in my book at least), it could allow us to leverage different audio reading/writing backends, which can be justifiable in different scenarios.

Ehhhhh i think we should just use pysoundfile for everything. Given everything else that it offers, I'm content to drop mp3 support.

[ejhumphrey]

Transforming audio without JAMS is also possible, in that you can have the jams pipe construct a dummy jams object.

yes, but you have to make sure you set jam.file_metadata.duration or things go to hell for time-stretching.

Ehhhhh i think we should just use pysoundfile for everything. Given everything else that it offers, I'm content to drop mp3 support.

At the risk of sparking a holy war, some scenarios justify using a sox or ffmpeg backend. If we're encapsulating things properly, I don't think it has to be gross.

[bmcfee]

yes, but you have to make sure you set jam.file_metadata.duration or things go to hell for time-stretching.

That's easy enough.

At the risk of sparking a holy war, some scenarios justify using a sox or ffmpeg backend.

I'm curious what those scenarios are. If we're writing audio, I feel okay forcing people to use ogg, flac, or wav, which don't require ffmpeg. The big win here comes down to dependency management -- I want to absolutely limit the number of non-python dependencies we have to deal with. Zero would be ideal.

Reading audio is another story, and if we want to support mp3, we're kinda stuck with the pile of hack that audioread abstracts away from us. We shouldn't reinvent that functionality. If we don't care so much about mp3, then pysoundfile all the way.

(I'm totally baffled as to why you would want to output to sox though.)

If we're encapsulating things properly, I don't think it has to be gross.

The implementation could be totally pristine, but the dependency management will bite us every time. I'm more than happy to let audioread and/or libsndfile handle codec business.

[bmcfee]

Closing this out, as I think the current containers are here to stay.