Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

ofQTKitPlayer audio monitoring functions #1979

Closed
wants to merge 7 commits into from

3 participants

@prisonerjohn

Added some functions for monitoring Quicktime movie audio frequency and volume levels.

You can configure the number of channels and bands, and you can set whether you want to monitor frequency or volume.

I also added an example that shows all this in action.

@obviousjim This is the same functionality that you looked at a couple of days ago, just a little more oF-y.

@kylemcdonald
Owner

@obviousjim what do you think about this?

@obviousjim
Collaborator

Thanks for the nudge. pulling into a local branch and taking a look,

@obviousjim
Collaborator

I looked through code and it seems really awesome, and definitely useful. Here are my concerns:

1) The QTKit based video player is straying further from the cross-platform supported features in OF. This is a general OF strategy, something that @arturoc could comment on. Is this a problem or is it ok?

2) Moving forward, I think we see QTKit as a bandaid solution for a GStreamer or AVFoundation implementation. Are we stitching ourselves up by adding this to the core in that we must 'promise' to support audio frequency with the similar API as we move forward to new solutions. Or can we leave it as a 'one off bonus' for those who choose to use ofQTKitPlayer directly?

From a code perspective, it looks clean and worked well for me. Be good to get a few more eyes on it, @bakercp always has a discretionary opinions

@prisonerjohn

Bump @arturoc @bakercp @obviousjim

I guess this could also be an addon if that makes more sense.

@obviousjim
Collaborator

Is there a way to structure it as an addon that just uses the ofQTKitPlayer object internally? I'd be worried about having another QTKit movie player with duplciated code floating around

@prisonerjohn

My first instinct would be to subclass ofQTKitPlayer and ofQTKitMovieRenderer and override ofQTKitPlayer::loadMovie() and `ofQTKitPlayer::update().

But that would require modifications to the current classes. I would need access to some private variables like ofQTKitPlayer.moviePlayer and [ofQTKitMovieRendere .movie].

@prisonerjohn prisonerjohn deleted the prisonerjohn:feature-qtkit-audio branch
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
View
BIN  examples/video/osxAudioVideoPlayerExample/bin/data/movies/cat.mp4
Binary file not shown
View
15 examples/video/osxAudioVideoPlayerExample/src/main.cpp
@@ -0,0 +1,15 @@
+#include "ofMain.h"
+#include "testApp.h"
+#include "ofAppGlutWindow.h"
+
+//========================================================================
+int main( ){
+ ofAppGlutWindow window;
+ ofSetupOpenGL(&window, 1024,768, OF_WINDOW); // <-------- setup the GL context
+
+ // this kicks off the running of my app
+ // can be OF_WINDOW or OF_FULLSCREEN
+ // pass in width and height too:
+ ofRunApp( new testApp());
+
+}
View
159 examples/video/osxAudioVideoPlayerExample/src/testApp.cpp
@@ -0,0 +1,159 @@
+#include "testApp.h"
+
+// OS X Audio Video Playback Example
+//--------------------------------------------------------------
+// This example shows how to use the OS X platform specific
+// ofQTKitPlayer on its own without the cross platform
+// ofVideoPlayer wrapper. Apps you write in this way won't be
+// cross platform, but will allow to access audio data.
+//--------------------------------------------------------------
+
+void testApp::setup(){
+ ofBackground(27);
+
+ // Enable audio frequency and volume metering.
+ catMovie.enableAudioFrequencyMetering(2);
+ catMovie.enableAudioVolumeMetering(2);
+
+ // Load and start the movie.
+ catMovie.loadMovie("movies/cat.mp4", OF_QTKIT_DECODE_PIXELS_AND_TEXTURE);
+ catMovie.play();
+}
+
+//--------------------------------------------------------------
+void testApp::update(){
+ catMovie.update();
+}
+
+//--------------------------------------------------------------
+void testApp::draw(){
+ if (catMovie.isLoaded()) {
+ int margin = 20;
+ int currTop = margin;
+
+ // Draw the movie.
+ ofSetColor(ofColor::white);
+ catMovie.draw((ofGetWidth() - catMovie.getWidth()) / 2, currTop);
+
+ currTop += catMovie.getHeight() + margin;
+
+ // Draw the volume levels.
+ // The levels returned are measured in deciBels, where 0.0 means full volume, -6.0 means half volume,
+ // -12.0 means quarter volume, and -inf means silence.
+ int volHeight = 20;
+ for (int i = 0; i < catMovie.getNumAudioVolumeChannels(); i++) {
+ ofFill();
+ ofSetColor(42);
+ ofRect(margin, currTop, 980, volHeight);
+
+ ofSetColor(255);
+ float volumeLevel = catMovie.getAudioVolumeLevel(i);
+ float volumeWidth;
+ if (volumeLevel >= 0.0)
+ volumeWidth = 980;
+ else if (volumeLevel >= -6.0)
+ volumeWidth = ofMap(volumeLevel, -6.0, 0.0, 490, 980);
+ else if (volumeLevel >= -12.0)
+ volumeWidth = ofMap(volumeLevel, -12.0, -6.0, 245, 490);
+ else
+ volumeWidth = ofMap(volumeLevel, -INFINITY, -12.0, 0, 245);
+ ofRect(margin, currTop, volumeWidth, volHeight);
+
+ ofSetColor(0);
+ ofNoFill();
+ ofRect(margin, currTop, 980, volHeight);
+
+ currTop += volHeight + 10;
+ }
+
+ currTop += 10;
+
+ ofSetColor(255);
+ ofDrawBitmapString("-inf", 10, currTop);
+ ofDrawBitmapString("-12 dB", 245, currTop);
+ ofDrawBitmapString("-6 dB", 485, currTop);
+ ofDrawBitmapString("0 dB", 985, currTop);
+
+ currTop += margin;
+
+ // Draw the audio frequency bands.
+ // The levels returned are normalized (0.0 to 1.0), but measured in Hertz.
+ int freqLeft = margin;
+ int freqWidth = 420;
+ int freqHeight = 380;
+ for (int i = 0; i < catMovie.getNumAudioFrequencyChannels(); i++) {
+ float bandTop = currTop;
+ float bandHeight = freqHeight / (float)catMovie.getNumAudioFrequencyBands();
+
+ ofFill();
+ ofSetColor(42);
+ ofRect(freqLeft, currTop, freqWidth, freqHeight);
+
+ for (int j = 0; j < catMovie.getNumAudioFrequencyBands(); j++) {
+ float bandWidth = MIN(1, catMovie.getAudioFrequencyLevel(i, j)) * freqWidth;
+
+ ofSetColor(255 - (12 * j));
+ ofRect(freqLeft, bandTop, bandWidth, bandHeight);
+
+ if (i == 0) {
+ ofSetColor(255);
+ ofDrawBitmapString(ofToString(catMovie.getAudioFrequencyMeteringBand(j), 1) + " Hz", 470, bandTop + 18);
+ }
+
+ bandTop += bandHeight;
+ }
+
+ ofNoFill();
+ ofSetColor(0);
+ ofRect(freqLeft, currTop, freqWidth, freqHeight);
+
+ freqLeft = 580;
+ }
+ }
+}
+
+//--------------------------------------------------------------
+void testApp::keyPressed(int key){
+
+}
+
+//--------------------------------------------------------------
+void testApp::keyReleased(int key){
+
+}
+
+//--------------------------------------------------------------
+void testApp::mouseMoved(int x, int y){
+
+}
+
+//--------------------------------------------------------------
+void testApp::mouseDragged(int x, int y, int button){
+
+}
+
+//--------------------------------------------------------------
+void testApp::mousePressed(int x, int y, int button){
+
+}
+
+
+//--------------------------------------------------------------
+void testApp::mouseReleased(int x, int y, int button){
+
+}
+
+//--------------------------------------------------------------
+void testApp::windowResized(int w, int h){
+
+}
+
+//--------------------------------------------------------------
+void testApp::gotMessage(ofMessage msg){
+
+}
+
+//--------------------------------------------------------------
+void testApp::dragEvent(ofDragInfo dragInfo){
+
+}
View
28 examples/video/osxAudioVideoPlayerExample/src/testApp.h
@@ -0,0 +1,28 @@
+#pragma once
+
+#include "ofMain.h"
+
+class testApp : public ofBaseApp {
+
+ public:
+
+ void setup();
+ void update();
+ void draw();
+
+ void keyPressed(int key);
+ void keyReleased(int key);
+ void mouseMoved(int x, int y);
+ void mouseDragged(int x, int y, int button);
+ void mousePressed(int x, int y, int button);
+ void mouseReleased(int x, int y, int button);
+ void windowResized(int w, int h);
+ void dragEvent(ofDragInfo dragInfo);
+ void gotMessage(ofMessage msg);
+
+ //instead of using ofVideoPlayer we use the ofQTKitPlayer directly
+ ofQTKitPlayer catMovie;
+
+ bool frameByframe;
+};
+
View
23 libs/openFrameworks/video/ofQTKitMovieRenderer.h
@@ -16,7 +16,17 @@
CVOpenGLTextureCacheRef _textureCache;
CVOpenGLTextureRef _latestTextureFrame;
CVPixelBufferRef _latestPixelFrame;
-
+
+ BOOL _audioFrequencyMeteringEnabled;
+// NSInteger _numAudioFrequencyChannels;
+// NSInteger _numAudioFrequencyBands;
+ FourCharCode _audioFrequencyMixToMeter;
+ BOOL _audioVolumeMeteringEnabled;
+// NSInteger _numAudioVolumeChannels;
+ QTAudioFrequencyLevels * _audioFrequencyLevels;
+ QTAudioVolumeLevels * _audioVolumeLevels;
+ Float32 * _audioFrequencyMeteringBandFrequencies;
+
NSSize movieSize;
QTTime movieDuration;
NSInteger frameCount;
@@ -46,6 +56,14 @@
@property (readwrite) BOOL justSetFrame; //this needs to be set *before* calls to _movie.setTime to allow synchronous seeking
@property (nonatomic, readwrite) BOOL synchronousSeek;
+@property (nonatomic, readonly) BOOL hasAudio;
+@property (nonatomic, readonly) BOOL hasVideo;
+
+@property (nonatomic, readonly) BOOL audioFrequencyMeteringEnabled;
+@property (nonatomic, readonly) QTAudioFrequencyLevels * audioFrequencyLevels;
+@property (nonatomic, readonly) Float32 * audioFrequencyMeteringBandFrequencies;
+@property (nonatomic, readonly) BOOL audioVolumeMeteringEnabled;
+@property (nonatomic, readonly) QTAudioVolumeLevels * audioVolumeLevels;
@property (nonatomic, readwrite) float rate;
@property (nonatomic, readwrite) float volume;
@@ -64,6 +82,8 @@
- (void)draw:(NSRect)drawRect;
- (BOOL)loadMovie:(NSString *)moviePath pathIsURL:(BOOL)isURL allowTexture:(BOOL)useTexture allowPixels:(BOOL)usePixels allowAlpha:(BOOL)useAlpha;
+- (void)enableAudioFrequencyMetering:(NSInteger)numChannels numBands:(NSInteger)numBands;
+- (void)enableAudioVolumeMetering:(NSInteger)numChannels;
- (BOOL)update;
- (void)bindTexture;
@@ -77,7 +97,6 @@
- (void)stepBackward;
- (void)gotoBeginning;
-
- (void)frameAvailable:(CVImageBufferRef)image;
- (void)frameFailed;
View
89 libs/openFrameworks/video/ofQTKitMovieRenderer.m
@@ -50,6 +50,26 @@ @implementation QTKitMovieRenderer
@synthesize justSetFrame;
@synthesize synchronousSeek;
+@synthesize hasAudio;
+@synthesize hasVideo;
+
+@synthesize audioFrequencyMeteringEnabled = _audioFrequencyMeteringEnabled;
+@synthesize audioFrequencyLevels = _audioFrequencyLevels;
+@synthesize audioFrequencyMeteringBandFrequencies = _audioFrequencyMeteringBandFrequencies;
+@synthesize audioVolumeMeteringEnabled = _audioVolumeMeteringEnabled;
+@synthesize audioVolumeLevels = _audioVolumeLevels;
+
+- (id)init
+{
+ self = [super init];
+ if (self) {
+ // set audio metering defaults
+ _audioFrequencyMeteringEnabled = NO;
+ _audioVolumeMeteringEnabled = NO;
+ }
+ return self;
+}
+
- (NSDictionary*) pixelBufferAttributes
{
return [NSDictionary dictionaryWithObjectsAndKeys:
@@ -190,9 +210,59 @@ - (BOOL) loadMovie:(NSString*)moviePath pathIsURL:(BOOL)isURL allowTexture:(BOOL
self.loops = YES;
self.palindrome = NO;
+ if (_audioFrequencyMeteringBandFrequencies != NULL) {
+ free(_audioFrequencyMeteringBandFrequencies);
+ _audioFrequencyMeteringBandFrequencies = NULL;
+ }
+ if (_audioFrequencyLevels != NULL) {
+ free(_audioFrequencyLevels);
+ _audioFrequencyLevels = NULL;
+ }
+ if (_audioVolumeLevels != NULL) {
+ free(_audioVolumeLevels);
+ _audioVolumeLevels = NULL;
+ }
+
return YES;
}
+- (void)enableAudioFrequencyMetering:(NSInteger)numChannels numBands:(NSInteger)numBands
+{
+ if (hasAudio == NO) return;
+
+ _audioFrequencyMeteringEnabled = YES;
+
+ _audioFrequencyLevels = (QTAudioFrequencyLevels *)malloc(offsetof(QTAudioFrequencyLevels, level[numBands * numChannels]));
+ _audioFrequencyLevels->numChannels = numChannels;
+ _audioFrequencyLevels->numFrequencyBands = numBands;
+ _audioFrequencyMeteringBandFrequencies = malloc(_audioFrequencyLevels->numFrequencyBands * sizeof(Float32));
+
+ if (_audioFrequencyLevels->numFrequencyBands == 1)
+ _audioFrequencyMixToMeter = kQTAudioMeter_MonoMix;
+ else if (_audioFrequencyLevels->numFrequencyBands == 2)
+ _audioFrequencyMixToMeter = kQTAudioMeter_StereoMix;
+ else
+ _audioFrequencyMixToMeter = kQTAudioMeter_DeviceMix;
+
+ SetMovieAudioFrequencyMeteringNumBands([_movie quickTimeMovie], _audioFrequencyMixToMeter, &_audioFrequencyLevels->numFrequencyBands);
+ OSStatus err = GetMovieAudioFrequencyMeteringBandFrequencies([_movie quickTimeMovie], _audioFrequencyMixToMeter, _audioFrequencyLevels->numFrequencyBands, _audioFrequencyMeteringBandFrequencies);
+
+ if (err)
+ NSLog(@"Error %ld getting movie audio frequency metering band frequencies", err);
+
+}
+
+- (void)enableAudioVolumeMetering:(NSInteger)numChannels
+{
+ if (hasAudio == NO) return;
+
+ _audioVolumeMeteringEnabled = YES;
+
+ _audioVolumeLevels = (QTAudioVolumeLevels *)malloc(offsetof(QTAudioVolumeLevels, level[numChannels]));
+ _audioVolumeLevels->numChannels = numChannels;
+ SetMovieAudioVolumeMeteringEnabled([_movie quickTimeMovie], kQTAudioMeter_DeviceMix, YES);
+}
+
- (void) dealloc
{
@synchronized(self){
@@ -235,6 +305,11 @@ - (void) dealloc
[synchronousSeekLock release];
synchronousSeekLock = nil;
}
+
+ if (hasAudio) {
+ free(_audioFrequencyLevels);
+ free(_audioVolumeLevels);
+ }
}
[super dealloc];
}
@@ -345,6 +420,20 @@ - (void)frameAvailable:(CVImageBufferRef)image
}
QTVisualContextTask(_visualContext);
+
+ if (hasAudio) {
+ OSStatus err = noErr;
+ if (_audioFrequencyMeteringEnabled) {
+ err = GetMovieAudioFrequencyLevels([_movie quickTimeMovie], _audioFrequencyMixToMeter, _audioFrequencyLevels);
+ if (err)
+ NSLog(@"Error %ld getting movie audio frequency levels", err);
+ }
+ if (_audioVolumeMeteringEnabled) {
+ err = GetMovieAudioVolumeLevels([_movie quickTimeMovie], kQTAudioMeter_DeviceMix, nil, _audioVolumeLevels);
+ if (err)
+ NSLog(@"Error %ld getting movie audio volume levels", err);
+ }
+ }
}
- (void)frameFailed
View
29 libs/openFrameworks/video/ofQTKitPlayer.h
@@ -1,10 +1,13 @@
// Copyright (c) 2012 openFrameworks team
// openFrameworks is released under the MIT License. See libs/_Licence.txt
+// Audio stuff based on http://www.okado.no/posts/how-to-use-itunes-music-visualizers-in-coco/index.html
+
#pragma once
#include "ofMain.h"
+#include <QuickTime/QuickTime.h>
#ifdef __OBJC__
#import "ofQTKitMovieRenderer.h"
#endif
@@ -98,6 +101,22 @@ class ofQTKitPlayer : public ofBaseVideoPlayer {
void setSynchronousSeeking(bool synchronous);
bool getSynchronousSeeking();
+ void enableAudioFrequencyMetering(int numChannels = 1, int numBands = 16);
+ void enableAudioVolumeMetering(int numChannels = 1);
+
+ void updateAudioMetering();
+
+ int getNumAudioFrequencyChannels();
+ int getNumAudioFrequencyBands();
+ vector<float>& getAudioFrequencyLevels();
+ float getAudioFrequencyLevel(int channel, int band);
+ vector<float>& getAudioFrequencyMeteringBands();
+ float getAudioFrequencyMeteringBand(int band);
+
+ int getNumAudioVolumeChannels();
+ vector<float>& getAudioVolumeLevels();
+ float getAudioVolumeLevel(int channel);
+
void draw(float x, float y, float w, float h);
void draw(float x, float y);
@@ -148,4 +167,14 @@ class ofQTKitPlayer : public ofBaseVideoPlayer {
void * moviePlayer;
#endif
+ int lastAudioUpdateFrame;
+ bool bAudioFrequencyMeteringEnabled;
+ int numAudioFrequencyChannels;
+ int numAudioFrequencyBands;
+ vector<float> audioFrequencyLevels;
+ vector<float> audioFrequencyMeteringBands;
+ bool bAudioVolumeMeteringEnabled;
+ int numAudioVolumeChannels;
+ vector<float> audioVolumeLevels;
+
};
View
130 libs/openFrameworks/video/ofQTKitPlayer.mm
@@ -12,7 +12,10 @@
speed = 1.0f;
// default this to true so the player update behavior matches ofQuicktimePlayer
bSynchronousSeek = true;
-
+
+ bAudioFrequencyMeteringEnabled = false;
+ bAudioVolumeMeteringEnabled = false;
+
pixelFormat = OF_PIXELS_RGB;
currentLoopState = OF_LOOP_NORMAL;
}
@@ -23,7 +26,109 @@
}
//--------------------------------------------------------------------
-bool ofQTKitPlayer::loadMovie(string path){
+void ofQTKitPlayer::enableAudioFrequencyMetering(int numChannels, int numBands) {
+ bAudioFrequencyMeteringEnabled = true;
+ numAudioFrequencyChannels = numChannels;
+ numAudioFrequencyBands = numBands;
+}
+
+//--------------------------------------------------------------------
+void ofQTKitPlayer::enableAudioVolumeMetering(int numChannels) {
+ bAudioVolumeMeteringEnabled = true;
+ numAudioVolumeChannels = numChannels;
+}
+
+//--------------------------------------------------------------------
+void ofQTKitPlayer::updateAudioMetering() {
+ if (moviePlayer == NULL)
+ return false;
+
+ if (moviePlayer.audioFrequencyMeteringEnabled) {
+ audioFrequencyLevels.clear();
+ for (int i = 0; i < moviePlayer.audioFrequencyLevels->numChannels; i++) {
+ for (int j = 0; j < moviePlayer.audioFrequencyLevels->numFrequencyBands; j++) {
+ int index = (i * moviePlayer.audioFrequencyLevels->numFrequencyBands) + j;
+ audioFrequencyLevels.push_back(moviePlayer.audioFrequencyLevels->level[index]);
+ }
+ }
+ }
+
+ if (moviePlayer.audioVolumeMeteringEnabled) {
+ audioVolumeLevels.clear();
+ for (int i = 0; i < moviePlayer.audioVolumeLevels->numChannels; i++) {
+ audioVolumeLevels.push_back(moviePlayer.audioVolumeLevels->level[i]);
+ }
+ }
+}
+
+//--------------------------------------------------------------------
+int ofQTKitPlayer::getNumAudioFrequencyChannels() {
+ if (moviePlayer && moviePlayer.audioFrequencyMeteringEnabled)
+ return moviePlayer.audioFrequencyLevels->numChannels;
+
+ return 0;
+}
+
+//--------------------------------------------------------------------
+int ofQTKitPlayer::getNumAudioFrequencyBands() {
+ if (moviePlayer && moviePlayer.audioFrequencyMeteringEnabled)
+ return moviePlayer.audioFrequencyLevels->numFrequencyBands;
+
+ return 0;
+}
+
+//--------------------------------------------------------------------
+vector<float>& ofQTKitPlayer::getAudioFrequencyLevels() {
+ return audioFrequencyLevels;
+}
+
+//--------------------------------------------------------------------
+float ofQTKitPlayer::getAudioFrequencyLevel(int channel, int band) {
+ if (moviePlayer && moviePlayer.audioFrequencyMeteringEnabled) {
+ int index = (channel * moviePlayer.audioFrequencyLevels->numFrequencyBands) + band;
+ if (audioFrequencyLevels.size() > index)
+ return audioFrequencyLevels[index];
+ }
+
+ return 0;
+}
+
+//--------------------------------------------------------------------
+vector<float>& ofQTKitPlayer::getAudioFrequencyMeteringBands() {
+ return audioFrequencyMeteringBands;
+}
+
+//--------------------------------------------------------------------
+float ofQTKitPlayer::getAudioFrequencyMeteringBand(int band) {
+ if (audioFrequencyMeteringBands.size() > band)
+ return audioFrequencyMeteringBands[band];
+
+ return 0;
+}
+
+//--------------------------------------------------------------------
+int ofQTKitPlayer::getNumAudioVolumeChannels() {
+ if (moviePlayer && moviePlayer.audioVolumeMeteringEnabled)
+ return moviePlayer.audioVolumeLevels->numChannels;
+
+ return 0;
+}
+
+//--------------------------------------------------------------------
+vector<float>& ofQTKitPlayer::getAudioVolumeLevels() {
+ return audioVolumeLevels;
+}
+
+//--------------------------------------------------------------------
+float ofQTKitPlayer::getAudioVolumeLevel(int channel) {
+ if (audioVolumeLevels.size() > channel)
+ return audioVolumeLevels[channel];
+
+ return -INFINITY;
+}
+
+//--------------------------------------------------------------------
+bool ofQTKitPlayer::loadMovie(string path) {
return loadMovie(path, OF_QTKIT_DECODE_PIXELS_ONLY);
}
@@ -72,6 +177,25 @@
setLoopState(currentLoopState);
setSpeed(1.0f);
firstFrame(); //will load the first frame
+
+ lastAudioUpdateFrame = -1;
+ if (bAudioFrequencyMeteringEnabled) {
+ [moviePlayer enableAudioFrequencyMetering:numAudioFrequencyChannels
+ numBands:numAudioFrequencyBands];
+
+ // store the actual audio frequency metering band levels (these won't change)
+ audioFrequencyMeteringBands.clear();
+ for (int i = 0; i < moviePlayer.audioFrequencyLevels->numChannels; i++) {
+ for (int j = 0; j < moviePlayer.audioFrequencyLevels->numFrequencyBands; j++) {
+ int index = (i * moviePlayer.audioFrequencyLevels->numFrequencyBands) + j;
+ audioFrequencyMeteringBands.push_back(moviePlayer.audioFrequencyMeteringBandFrequencies[index]);
+ }
+ }
+ }
+
+ if (bAudioVolumeMeteringEnabled) {
+ [moviePlayer enableAudioVolumeMetering:numAudioVolumeChannels];
+ }
}
else {
ofLogError("ofQTKitPlayer") << "Loading file " << movieFilePath << " failed.";
@@ -210,6 +334,8 @@
bNewFrame = [moviePlayer update];
if (bNewFrame) {
bHavePixelsChanged = true;
+
+ updateAudioMetering();
}
[pool release];
}
Something went wrong with that request. Please try again.