Document audio sample formats. [Replaced]

[fommil]

Apologies for asking this in the issue tracker, I didn't see any forum linked from the README.

I have an av.AudioFrame

<av.AudioFrame 0, 1024 samples at 48000Hz, mono, fltp at 0x7ffa7f939ec8>

but calling .to_nd_array() on it gives me this error

(Pdb) frame.to_nd_array()
*** AssertionError: ("Don't know how to convert data type", 'fltp')

how can I see the samples?

[fommil]

this may well be related to #70 but I'm afraid I have no idea how to access the "buffer interface"

[fommil]

Is this right? np.frombuffer(frame.planes[0].to_bytes(), dtype=np.uint8)

It certainly gives me an array of numbers, but I'm not sure how to interpret it.

Ultimately, I want to compute a DTFT from a video. But I don't necessarily want to be tied to the frame size for my time spacing. So ideally, for maximum flexibility, I'd like to be able to generate the full raw audio track.

[mikeboers]

Take a look at https://www.ffmpeg.org/doxygen/2.8/group__lavu__sampfmts.html#gaf9a51ca15301871723577c730b5865c5 to see what the formats mean. "fltp" will be planar float, so one plane per channel, as floats.

You should also be able to use the buffer interface directly, so I'd try:

np.frombuffer(frame.planes[0], dtype=float)

(I did not try the above.)

I'll leave this open as a signal that this should be documented better.

[fommil]

ok cool, thanks!

I now have a big bunch of arrays of floats... currently trying to understand how to convert this into something play or aplay will accept so I can hear that it is the right thing.

[fommil]

play -t raw -r 48k -e floating-point -b 32 -c 1 audio.raw

[fommil]

I'm not convinced that the data is what I think it is. Plotting it

        container = av.open(datum.f)
        frames = container.streams.audio[0].frames
        chunk = container.streams.audio[0].frame_size // 2  # bug in av?
        rate = container.streams.audio[0].rate
        fltp = np.zeros((frames, chunk), dtype=float)
        for n, frame in enumerate(container.decode(audio=0)):
            fltp[n, :] = np.frombuffer(frame.planes[0], dtype=float)
        fltp = fltp.flatten()
        plt.plot(fltp)
        plt.show()

has some huge volume spikes.

I'm having a hell of a time trying to create a FFT out of this data https://stackoverflow.com/questions/44528927

[fommil]

doh! np.float32

[mikeboers]

Oh yeah... Python's float is 64bit. Whoops. 😜

[fommil]

people keep telling me Python is "easy" and fast to write, and Scala / Haskell is "hard" and slow to write code, and I keep saying "well, this week I lost X days to a simple type error, and it didn't even involve higher order types".

[tritium21]

To be completely fair, the problem wasn't python's float types, it was fortran's float types. Numpy is a fairly thick wrapper around blas...