Skip to content
Permalink
Browse files

Bug 1094764 - Implement AudioContext.suspend and friends. r=roc,ehsan

- Relevant spec text:
    - http://webaudio.github.io/web-audio-api/#widl-AudioContext-suspend-Promise
    - http://webaudio.github.io/web-audio-api/#widl-AudioContext-resume-Promise
    - http://webaudio.github.io/web-audio-api/#widl-AudioContext-close-Promise
    - http://webaudio.github.io/web-audio-api/#widl-AudioContext-state
    - http://webaudio.github.io/web-audio-api/#widl-AudioContext-onstatechange

- In a couple words, the behavior we want:
    - Closed context cannot have new nodes created, but can do decodeAudioData,
    and create buffers, and such.
    - OfflineAudioContexts don't support those methods, transitions happen at
    startRendering and at the end of processing. onstatechange is used to make
    this observable.
    - (regular) AudioContexts support those methods. The promises and
    onstatechange should be resolved/called when the operation has actually
    completed on the rendering thread.  Once a context has been closed, it
    cannot transition back to "running". An AudioContext switches to "running"
    when the audio callback start running, this allow authors to know how long
    the audio stack takes to start running.
    - MediaStreams that feed in/go out of a suspended graph should respectively
    not buffer at the graph input, and output silence
    - suspended context should not be doing much on the CPU, and we should try
    to pause audio streams if we can (this behaviour is the main reason we need
    this in the first place, for saving battery on mobile, and CPU on all
    platforms)

- Now, the implementation:
    - AudioNodeStreams are now tagged with a context id, to be able to operate
    on all the streams of a given AudioContext on the Graph thread without
    having to go and lock everytime to touch the AudioContext. This happens in
    the AudioNodeStream ctor. IDs are of course constant for the lifetime of the
    node.
    - When an AudioContext goes into suspended mode, streams for this
    AudioContext are moved out of the mStreams array to a second array,
    mSuspendedStreams. Streams in mSuspendedStream are not ordered, and are not
    processed.
    - The MSG will automatically switch to a SystemClockDriver when it finds
    that there are no more AudioNodeStream/Stream with an audio track. This is
    how pausing the audio subsystem and saving battery works. Subsequently, when
    the MSG finds that there are only streams in mSuspendedStreams, it will go
    to sleep (block on a monitor), so we save CPU, but it does not shut itself
    down. This is mostly not a new behaviour (this is what the MSG does since
    the refactoring), but is important to note.
    - Promises are gripped (addref-ed) on the main thread, and then shepherd
    down other threads and to the GraphDriver, if needed (sometimes we can
    resolve them right away). They move between threads as void* to prevent
    calling methods on them, as they are not thread safe. Then, the driver
    executes the operation, and when it's done (initializing and closing audio
    streams can take some time), we send the promise back to the main thread,
    and resolve it, casting back to Promise* after asserting we're back on the
    main thread. This way, we can send them back on the main thread once an
    operation has complete (suspending an audio stream, starting it again on
    resume(), etc.), without having to do bookkeeping between suspend calls and
    their result. Promises are not thread safe, so we can't move them around
    AddRef-ed.
    - The stream destruction logic now takes into account that a stream can be
    destroyed while not being in mStreams.
    - A graph can now switch GraphDriver twice or more per iteration, for
    example if an author goes suspend()/resume()/suspend() in the same script.
    - Some operation have to be done on suspended stream, so we now use double
    for-loop around mSuspendedStreams and mStreams in some places in
    MediaStreamGraph.cpp.
    - A tricky part was making sure everything worked at AudioContext
    boundaries.  TrackUnionStream that have one of their input stream suspended
    append null ticks instead.
    - The graph ordering algorithm had to be altered to not include suspended
    streams.
    - There are some edge cases (adding a stream on a suspended graph, calling
    suspend/resume when a graph has just been close()d).
  • Loading branch information...
rmottola committed Jul 2, 2019
1 parent f2c3721 commit 79ff7e5eaa14243bf688420fe6516735f360d036
@@ -13178,7 +13178,8 @@ nsGlobalWindow::SuspendTimeouts(uint32_t aIncrease,

// Suspend all of the AudioContexts for this window
for (uint32_t i = 0; i < mAudioContexts.Length(); ++i) {
mAudioContexts[i]->Suspend();
ErrorResult dummy;
unused << mAudioContexts[i]->Suspend(dummy);
}
}

@@ -13238,7 +13239,8 @@ nsGlobalWindow::ResumeTimeouts(bool aThawChildren)

// Resume all of the AudioContexts for this window
for (uint32_t i = 0; i < mAudioContexts.Length(); ++i) {
mAudioContexts[i]->Resume();
ErrorResult dummy;
unused << mAudioContexts[i]->Resume(dummy);
}

// Thaw all of the workers for this window.
@@ -23,7 +23,7 @@ extern PRLogModuleInfo* gMediaStreamGraphLog;
#ifdef ENABLE_LIFECYCLE_LOG
#ifdef ANDROID
#include "android/log.h"
#define LIFECYCLE_LOG(args...) __android_log_print(ANDROID_LOG_INFO, "Goanna - MSG" , ## __VA_ARGS__); printf(__VA_ARGS__);printf("\n");
#define LIFECYCLE_LOG(...) __android_log_print(ANDROID_LOG_INFO, "Goanna - MSG" , __VA_ARGS__); printf(__VA_ARGS__);printf("\n");
#else
#define LIFECYCLE_LOG(...) printf(__VA_ARGS__);printf("\n");
#endif
@@ -95,9 +95,6 @@ void GraphDriver::SwitchAtNextIteration(GraphDriver* aNextDriver)
LIFECYCLE_LOG("Switching to new driver: %p (%s)",
aNextDriver, aNextDriver->AsAudioCallbackDriver() ?
"AudioCallbackDriver" : "SystemClockDriver");
// Sometimes we switch twice to a new driver per iteration, this is probably a
// bug.
MOZ_ASSERT(!mNextDriver || mNextDriver->AsAudioCallbackDriver());
mNextDriver = aNextDriver;
}

@@ -145,7 +142,7 @@ class MediaStreamGraphShutdownThreadRunnable : public nsRunnable {
LIFECYCLE_LOG("Releasing audio driver off main thread.");
nsRefPtr<AsyncCubebTask> releaseEvent =
new AsyncCubebTask(mDriver->AsAudioCallbackDriver(),
AsyncCubebTask::SHUTDOWN);
AsyncCubebOperation::SHUTDOWN);
mDriver = nullptr;
releaseEvent->Dispatch();
} else {
@@ -163,7 +160,7 @@ void GraphDriver::Shutdown()
if (AsAudioCallbackDriver()) {
LIFECYCLE_LOG("Releasing audio driver off main thread (GraphDriver::Shutdown).\n");
nsRefPtr<AsyncCubebTask> releaseEvent =
new AsyncCubebTask(AsAudioCallbackDriver(), AsyncCubebTask::SHUTDOWN);
new AsyncCubebTask(AsAudioCallbackDriver(), AsyncCubebOperation::SHUTDOWN);
releaseEvent->Dispatch();
} else {
Stop();
@@ -204,7 +201,7 @@ class MediaStreamGraphInitThreadRunnable : public nsRunnable {
// because the osx audio stack is currently switching output device.
if (!mDriver->mPreviousDriver->AsAudioCallbackDriver()->IsSwitchingDevice()) {
nsRefPtr<AsyncCubebTask> releaseEvent =
new AsyncCubebTask(mDriver->mPreviousDriver->AsAudioCallbackDriver(), AsyncCubebTask::SHUTDOWN);
new AsyncCubebTask(mDriver->mPreviousDriver->AsAudioCallbackDriver(), AsyncCubebOperation::SHUTDOWN);
mDriver->mPreviousDriver = nullptr;
releaseEvent->Dispatch();
}
@@ -505,36 +502,21 @@ AsyncCubebTask::Run()
MOZ_ASSERT(mDriver);

switch(mOperation) {
case AsyncCubebOperation::INIT:
case AsyncCubebOperation::INIT: {
LIFECYCLE_LOG("AsyncCubebOperation::INIT\n");
mDriver->Init();
mDriver->CompleteAudioContextOperations(mOperation);
break;
case AsyncCubebOperation::SHUTDOWN:
}
case AsyncCubebOperation::SHUTDOWN: {
LIFECYCLE_LOG("AsyncCubebOperation::SHUTDOWN\n");
mDriver->Stop();

mDriver->CompleteAudioContextOperations(mOperation);

mDriver = nullptr;
mShutdownGrip = nullptr;
break;
case AsyncCubebOperation::SLEEP: {
{
LIFECYCLE_LOG("AsyncCubebOperation::SLEEP\n");
MonitorAutoLock mon(mDriver->mGraphImpl->GetMonitor());
// We might just have been awoken
if (mDriver->mGraphImpl->mNeedAnotherIteration) {
mDriver->mPauseRequested = false;
mDriver->mWaitState = AudioCallbackDriver::WAITSTATE_RUNNING;
mDriver->mGraphImpl->mGraphDriverAsleep = false ; // atomic
break;
}
mDriver->Stop();
mDriver->mGraphImpl->mGraphDriverAsleep = true; // atomic
mDriver->mWaitState = AudioCallbackDriver::WAITSTATE_WAITING_INDEFINITELY;
mDriver->mPauseRequested = false;
mDriver->mGraphImpl->GetMonitor().Wait(PR_INTERVAL_NO_TIMEOUT);
}
STREAM_LOG(PR_LOG_DEBUG, ("Restarting audio stream from sleep."));
mDriver->StartStream();
break;
}
default:
MOZ_CRASH("Operation not implemented.");
@@ -546,6 +528,16 @@ AsyncCubebTask::Run()
return NS_OK;
}

StreamAndPromiseForOperation::StreamAndPromiseForOperation(MediaStream* aStream,
void* aPromise,
dom::AudioContextOperation aOperation)
: mStream(aStream)
, mPromise(aPromise)
, mOperation(aOperation)
{
MOZ_ASSERT(aPromise);
}

AudioCallbackDriver::AudioCallbackDriver(MediaStreamGraphImpl* aGraphImpl, dom::AudioChannel aChannel)
: GraphDriver(aGraphImpl)
, mIterationDurationMS(MEDIA_GRAPH_TARGET_PERIOD_MS)
@@ -561,7 +553,9 @@ AudioCallbackDriver::AudioCallbackDriver(MediaStreamGraphImpl* aGraphImpl, dom::
}

AudioCallbackDriver::~AudioCallbackDriver()
{}
{
MOZ_ASSERT(mPromisesForOperation.IsEmpty());
}

void
AudioCallbackDriver::Init()
@@ -651,12 +645,18 @@ AudioCallbackDriver::Start()
if (NS_IsMainThread()) {
STREAM_LOG(PR_LOG_DEBUG, ("Starting audio threads for MediaStreamGraph %p from a new thread.", mGraphImpl));
nsRefPtr<AsyncCubebTask> initEvent =
new AsyncCubebTask(this, AsyncCubebTask::INIT);
new AsyncCubebTask(this, AsyncCubebOperation::INIT);
initEvent->Dispatch();
} else {
STREAM_LOG(PR_LOG_DEBUG, ("Starting audio threads for MediaStreamGraph %p from the previous driver's thread", mGraphImpl));
Init();

// Check if we need to resolve promises because the driver just got switched
// because of a resuming AudioContext
if (!mPromisesForOperation.IsEmpty()) {
CompleteAudioContextOperations(AsyncCubebOperation::INIT);
}

if (mPreviousDriver) {
nsCOMPtr<nsIRunnable> event =
new MediaStreamGraphShutdownThreadRunnable(mPreviousDriver);
@@ -704,7 +704,7 @@ AudioCallbackDriver::Revive()
} else {
STREAM_LOG(PR_LOG_DEBUG, ("Starting audio threads for MediaStreamGraph %p from a new thread.", mGraphImpl));
nsRefPtr<AsyncCubebTask> initEvent =
new AsyncCubebTask(this, AsyncCubebTask::INIT);
new AsyncCubebTask(this, AsyncCubebOperation::INIT);
initEvent->Dispatch();
}
}
@@ -729,20 +729,6 @@ AudioCallbackDriver::GetCurrentTime()

void AudioCallbackDriver::WaitForNextIteration()
{
#if 0
mGraphImpl->GetMonitor().AssertCurrentThreadOwns();

// We can't block on the monitor in the audio callback, so we kick off a new
// thread that will pause the audio stream, and restart it when unblocked.
// We don't want to sleep when we haven't started the driver yet.
if (!mGraphImpl->mNeedAnotherIteration && mAudioStream && mGraphImpl->Running()) {
STREAM_LOG(PR_LOG_DEBUG+1, ("AudioCallbackDriver going to sleep"));
mPauseRequested = true;
nsRefPtr<AsyncCubebTask> sleepEvent =
new AsyncCubebTask(this, AsyncCubebTask::SLEEP);
sleepEvent->Dispatch();
}
#endif
}

void
@@ -1074,5 +1060,47 @@ AudioCallbackDriver::IsStarted() {
return mStarted;
}

void
AudioCallbackDriver::EnqueueStreamAndPromiseForOperation(MediaStream* aStream,
void* aPromise,
dom::AudioContextOperation aOperation)
{
MonitorAutoLock mon(mGraphImpl->GetMonitor());
mPromisesForOperation.AppendElement(StreamAndPromiseForOperation(aStream,
aPromise,
aOperation));
}

void AudioCallbackDriver::CompleteAudioContextOperations(AsyncCubebOperation aOperation)
{
nsAutoTArray<StreamAndPromiseForOperation, 1> array;

// We can't lock for the whole function because AudioContextOperationCompleted
// will grab the monitor
{
MonitorAutoLock mon(GraphImpl()->GetMonitor());
array.SwapElements(mPromisesForOperation);
}

for (int32_t i = array.Length() - 1; i >= 0; i--) {
StreamAndPromiseForOperation& s = array[i];
if ((aOperation == AsyncCubebOperation::INIT &&
s.mOperation == AudioContextOperation::Resume) ||
(aOperation == AsyncCubebOperation::SHUTDOWN &&
s.mOperation != AudioContextOperation::Resume)) {

GraphImpl()->AudioContextOperationCompleted(s.mStream,
s.mPromise,
s.mOperation);
array.RemoveElementAt(i);
}
}

if (!array.IsEmpty()) {
MonitorAutoLock mon(GraphImpl()->GetMonitor());
mPromisesForOperation.AppendElements(array);
}
}


} // namepace mozilla
@@ -13,6 +13,7 @@
#include "AudioSegment.h"
#include "SelfRef.h"
#include "mozilla/Atomics.h"
#include "AudioContext.h"

struct cubeb_stream;

@@ -321,6 +322,21 @@ class OfflineClockDriver : public ThreadedDriver
GraphTime mSlice;
};

struct StreamAndPromiseForOperation
{
StreamAndPromiseForOperation(MediaStream* aStream,
void* aPromise,
dom::AudioContextOperation aOperation);
nsRefPtr<MediaStream> mStream;
void* mPromise;
dom::AudioContextOperation mOperation;
};

enum AsyncCubebOperation {
INIT,
SHUTDOWN
};

/**
* This is a graph driver that is based on callback functions called by the
* audio api. This ensures minimal audio latency, because it means there is no
@@ -392,6 +408,12 @@ class AudioCallbackDriver : public GraphDriver,
return this;
}

/* Enqueue a promise that is going to be resolved when a specific operation
* occurs on the cubeb stream. */
void EnqueueStreamAndPromiseForOperation(MediaStream* aStream,
void* aPromise,
dom::AudioContextOperation aOperation);

bool IsSwitchingDevice() {
#ifdef XP_MACOSX
return mSelfReference;
@@ -414,6 +436,8 @@ class AudioCallbackDriver : public GraphDriver,
/* Tell the driver whether this process is using a microphone or not. This is
* thread safe. */
void SetMicrophoneActive(bool aActive);

void CompleteAudioContextOperations(AsyncCubebOperation aOperation);
private:
/**
* On certain MacBookPro, the microphone is located near the left speaker.
@@ -471,6 +495,7 @@ class AudioCallbackDriver : public GraphDriver,
/* Thread for off-main-thread initialization and
* shutdown of the audio stream. */
nsCOMPtr<nsIThread> mInitShutdownThread;
nsAutoTArray<StreamAndPromiseForOperation, 1> mPromisesForOperation;
dom::AudioChannel mAudioChannel;
Atomic<bool> mInCallback;
/* A thread has been created to be able to pause and restart the audio thread,
@@ -498,12 +523,6 @@ class AudioCallbackDriver : public GraphDriver,
class AsyncCubebTask : public nsRunnable
{
public:
enum AsyncCubebOperation {
INIT,
SHUTDOWN,
SLEEP
};


AsyncCubebTask(AudioCallbackDriver* aDriver, AsyncCubebOperation aOperation);

0 comments on commit 79ff7e5

Please sign in to comment.
You can’t perform that action at this time.