From 7ee9bbddd86b6ca0de7fc1bf7e862cbfa16ea92b Mon Sep 17 00:00:00 2001 From: Ted Moore Date: Wed, 11 May 2022 18:11:31 +0100 Subject: [PATCH 1/3] BufSTFT RST and SC example code --- doc/BufSTFT.rst | 14 ++++---- example-code/sc/BufSTFT.scd | 72 ++++++++++++++++++++++--------------- 2 files changed, 51 insertions(+), 35 deletions(-) diff --git a/doc/BufSTFT.rst b/doc/BufSTFT.rst index 7a37c80..5bd487c 100644 --- a/doc/BufSTFT.rst +++ b/doc/BufSTFT.rst @@ -4,15 +4,16 @@ :sc-related: Classes/Buffer :see-also: :description: - Performs either a forward or inverse Short-Time Fourier Transform (STFT) on a single channel source |buffer|. In the forward case, resulting magnitudes and phases can be written to output buffers. In the inverse case, these buffers can be used to reconstruct the original source into a new buffer. - The magntude and phase buffers are laid out as (number of hops, number of bins). The number of hops is a function of the source length and the hop size. The number of bins is (1 + (fft size / 2)). + Performs either a forward or inverse Short-Time Fourier Transform (STFT) on a single channel ``source`` |buffer|. In the forward case, resulting magnitudes and phases can be written to output buffers. In the inverse case, these buffers can be used to reconstruct the original ``source`` into a new buffer. - The object is restricted to analysing a single source channel, because the channel counts of the magntude and phase buffers would quickly get out of hand otherwise. + The ``magnitude`` and ``phase`` buffers are laid out so the frames are the number of analysis windows in the ``source`` buffer while the channels are the different bins. The number of hops is a function of the ``source`` length and the ``hopSize``. The number of bins is **(1 + (``fftSize`` / 2))**. + + The object is restricted to analysing a single source channel, because the channel counts of the ``magnitude`` and ``phase`` buffers would quickly get out of hand otherwise. .. only_in:: sc - If using an fftSize > 1024 the number of channels in the magnitude and phase buffers will be > 1024, which is the maximum number of channels a buffer can have when using |buffer|'s instance method ``loadToFloatArray``. This means you won't be able to get the values from the buffer using ``loadToFloatArray``. Instead you can use |buffer|'s instance method ``getToFloatArray``. + If using an ``fftSize`` > 1024 the number of channels in the ``magnitude`` and ``phase`` buffers will be > 1024, which is the maximum number of channels a buffer can have when using |buffer|'s instance method ``loadToFloatArray``. This means you won't be able to get the values from the buffer using ``loadToFloatArray``. Instead you can use |buffer|'s instance method ``getToFloatArray``. :control source: @@ -52,13 +53,12 @@ :control hopSize: - How many samples there are in-between analysis windows. The -1 default value will default to half of windowSize (overlap of 2). + How many samples there are in-between the start position of the analysis windows. The -1 default value will default to half of windowSize (overlap of 2). :control fftSize: - The FFT/IFFT size. It should be at least 4 samples long, at least the size of the window, and a power of 2. Making it larger allows an oversampling of the spectral precision. The -1 default value will use the next power of 2 equal or above the windowSize. For this object it is effectively capped at 65536. + The FFT/IFFT size. It should be at least 4 samples long, at least the size of the window, and a power of 2. Making it larger allows an oversampling of the spectral precision. The -1 default value will use the next power of 2 equal to or above the windowSize. For this object it is effectively capped at 65536. :control padding: Controls the zero-padding added to either end of the source buffer or segment. Possible values are 0 (no padding), 1 (default, half the window size), or 2 (window size - hop size). Padding ensures that all input samples are completely analysed: with no padding, the first analysis window starts at time 0, and the samples at either end will be tapered by the STFT windowing function. Mode 1 has the effect of centering the first sample in the analysis window and ensuring that the very start and end of the segment are accounted for in the analysis. Mode 2 can be useful when the overlap factor (window size / hop size) is greater than 2, to ensure that the input samples at either end of the segment are covered by the same number of analysis frames as the rest of the analysed material. - diff --git a/example-code/sc/BufSTFT.scd b/example-code/sc/BufSTFT.scd index 13bca19..b960580 100644 --- a/example-code/sc/BufSTFT.scd +++ b/example-code/sc/BufSTFT.scd @@ -1,41 +1,57 @@ code:: -s.reboot + +s.boot; + +~src = Buffer.read(s,FluidFilesPath("Olencki-TenTromboneLongTones-M.wav")); + +~src.play; + +// see the spectrogram of magnitudes ( -b = Buffer.read(s,FluidFilesPath("Nicol-LoopE-M.wav")); -m = Buffer.new; -p = Buffer.new; -r = Buffer.new; +~mags = Buffer(s); +~phases = Buffer(s); +FluidBufSTFT.processBlocking(s,~src,magnitude:~mags,phase:~phases,action:{ + defer{ + FluidWaveform(imageBuffer:~mags,imageColorScheme:1,imageColorScaling:1); + }; +}); ) +// the phases are less useful perhaps... ( -fork{ - FluidBufSTFT.process(s,source:b,magnitude:m,phase:p).wait; - FluidBufSTFT.process(s,magnitude:m,phase:p,resynth:r,inverse:1).wait; - "Done".postln; -} +FluidWaveform(imageBuffer:~phases,imageColorScheme:1,imageColorScaling:0); ) -{ PlayBuf.ar(1,r); }.play +// transform the magnitudes in the frequency domain for the spectral adjustments +( +~mags = Buffer(s); +~phases = Buffer(s); +FluidBufSTFT.processBlocking(s,~src,magnitude:~mags,phase:~phases); +~mags.loadToFloatArray(action:{ + arg mags; + mags = mags.clump(~mags.numChannels); // clump into analysis frames + mags = mags.collect{ // collect all the analysis frames + arg frame, idx; + frame.collect{ + // in each frame, multiply each value by a multiplier (the nested math here is causing + // it to do some modulation over time as well as frequency... take a listen + arg mag, bin; + mag * ((bin+(idx * 0.7)) * 0.01 * pi).sin.abs.pow(3); + }; + }; + ~mags.loadCollection(mags.flat,action:{ -//nullsum -{ PlayBuf.ar(1,r) - PlayBuf(1,b); }.play + ~resynth = Buffer(s); + FluidBufSTFT.processBlocking(s,magnitude:~mags,phase:~phases,resynth:~resynth,inverse:1,action:{ -//draw the magnitudes as a greyscale spectrogram -// make the image -i = Image.new(m.numFrames, m.numChannels) + "playing resynth".postln; -//retreive the image and assign to pixels -( -m.loadToFloatArray(action: {|x| - var mod = m.numChannels; - { - x.do{ - |val, index| - i.setColor(Color.gray(val), index.div(mod), mod - 1 - index.mod(mod)); - }; - i.plot("spectrogram", showInfo: false); - }.fork(AppClock) + ~resynth.postln; + defer{FluidWaveform(~resynth)}; + ~resynth.play; + }); + }); }); ) -:: +:: \ No newline at end of file From b5c9e5b897232f11d007ca93bd9b73a881b6f95e Mon Sep 17 00:00:00 2001 From: Ted Moore Date: Wed, 25 May 2022 15:54:04 +0100 Subject: [PATCH 2/3] added discussion --- doc/BufSTFT.rst | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/doc/BufSTFT.rst b/doc/BufSTFT.rst index 5bd487c..ba2afd3 100644 --- a/doc/BufSTFT.rst +++ b/doc/BufSTFT.rst @@ -5,7 +5,11 @@ :see-also: :description: - Performs either a forward or inverse Short-Time Fourier Transform (STFT) on a single channel ``source`` |buffer|. In the forward case, resulting magnitudes and phases can be written to output buffers. In the inverse case, these buffers can be used to reconstruct the original ``source`` into a new buffer. + Performs either a forward or inverse Short-Time Fourier Transform (STFT) on a single channel ``source`` |buffer|. + +:discussion: + + In the forward case, resulting magnitudes and phases can be written to output buffers. In the inverse case, these buffers can be used to reconstruct the original ``source`` into a new buffer. The ``magnitude`` and ``phase`` buffers are laid out so the frames are the number of analysis windows in the ``source`` buffer while the channels are the different bins. The number of hops is a function of the ``source`` length and the ``hopSize``. The number of bins is **(1 + (``fftSize`` / 2))**. From b4632ae25ec4783dcbd7c9bee6b722629269d413 Mon Sep 17 00:00:00 2001 From: Ted Moore Date: Wed, 1 Jun 2022 12:36:32 +0100 Subject: [PATCH 3/3] typo --- doc/BufSTFT.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/BufSTFT.rst b/doc/BufSTFT.rst index ba2afd3..7a7c7e4 100644 --- a/doc/BufSTFT.rst +++ b/doc/BufSTFT.rst @@ -43,7 +43,7 @@ The |buffer| to write phases to in the forward case, or read from in the inverse case. This is optional for the forward transform, mandatory for the inverse. -:control resynthesis: +:control resynth: The |buffer| to write re-synthesised data to in the inverse case. Ignored for the forward transform. Mandatory in the inverse case.