diff --git a/README.md b/README.md index 1193be90..18c4725c 100644 --- a/README.md +++ b/README.md @@ -28,22 +28,27 @@ Transcoder.into(filePath) Take a look at the demo app for a real example or keep reading below for documentation. -*Note: this project is an improved fork of [ypresto/android-transcoder](https://github.com/ypresto/android-transcoder). -It features a lot of improvements over the original project, including:* - -- *Multithreading support* -- *Crop to any aspect ratio* -- *Set output video rotation* -- *Various bugs fixed* -- *[Input](#data-sources): Accept content Uris and other types* -- *[Real error handling](#listening-for-events) instead of errors being thrown* -- *Frame dropping support, which means you can set the video frame rate* -- *Source project is over-conservative when choosing options that *might* not be supported. We prefer to try and let the codec fail* -- *More convenient APIs for transcoding & choosing options* -- *Configurable [Validators](#validators) to e.g. **not** perform transcoding if the source video is already compressed enough* -- *Expose internal logs through Logger (so they can be reported to e.g. Crashlytics)* -- *Handy utilities for track configuration through [Output Strategies](#output-strategies)* -- *Handy utilities for resizing* +## Features + +- Fast transcoding to AAC/AVC +- Hardware accelerated +- Multithreaded +- Convenient, fluent API +- Choose output size, with automatic cropping [[docs]](#video-size) +- Choose output rotation [[docs]](#video-rotation) +- Choose output speed [[docs]](#video-speed) +- Choose output frame rate [[docs]](#other-options) +- Choose output audio channels [[docs]](#audio-strategies) +- Override frames timestamp, e.g. to slow down the middle part of the video [[docs]](#time-interpolation) +- Error handling [[docs]](#listening-for-events) +- Configurable validators to e.g. avoid transcoding if the source is already compressed enough [[docs]](#validators) +- Configurable video and audio strategies [[docs]](#output-strategies) + +*This project started as a fork of [ypresto/android-transcoder](https://github.com/ypresto/android-transcoder). +With respect to the source project, which misses most of the functionality listed above, +we have also fixed a huge number of bugs and are much less conservative when choosing options +that might not be supported. The source project will always throw - for example, accepting only 16:9, +AVC Baseline Profile videos - we prefer to try and let the codec fail if it wants to*. ## Setup @@ -128,8 +133,8 @@ Transcoding operation did succeed. The success code can be: |Code|Meaning| |----|-------| -|`MediaTranscoder.SUCCESS_TRANSCODED`|Transcoding was executed successfully. Transcoded file was written to the output path.| -|`MediaTranscoder.SUCCESS_NOT_NEEDED`|Transcoding was not executed because it was considered **not needed** by the `Validator`.| +|`Transcoder.SUCCESS_TRANSCODED`|Transcoding was executed successfully. Transcoded file was written to the output path.| +|`Transcoder.SUCCESS_NOT_NEEDED`|Transcoding was not executed because it was considered **not needed** by the `Validator`.| Keep reading [below](#validators) to know about `Validator`s. @@ -220,9 +225,9 @@ audio stream to AAC format with the specified number of channels. ```java Transcoder.into(filePath) - .setAudioOutputStrategy(DefaultAudioStrategy(1)) // or.. - .setAudioOutputStrategy(DefaultAudioStrategy(2)) // or.. - .setAudioOutputStrategy(DefaultAudioStrategy(DefaultAudioStrategy.AUDIO_CHANNELS_AS_IS)) + .setAudioOutputStrategy(new DefaultAudioStrategy(1)) // or.. + .setAudioOutputStrategy(new DefaultAudioStrategy(2)) // or.. + .setAudioOutputStrategy(new DefaultAudioStrategy(DefaultAudioStrategy.AUDIO_CHANNELS_AS_IS)) // ... ``` @@ -243,16 +248,16 @@ We provide helpers for common tasks: DefaultVideoStrategy strategy; // Sets an exact size. If aspect ratio does not match, cropping will take place. -strategy = DefaultVideoStrategy.exact(1080, 720).build() +strategy = DefaultVideoStrategy.exact(1080, 720).build(); // Keeps the aspect ratio, but scales down the input size with the given fraction. -strategy = DefaultVideoStrategy.fraction(0.5F).build() +strategy = DefaultVideoStrategy.fraction(0.5F).build(); // Ensures that each video size is at most the given value - scales down otherwise. -strategy = DefaultVideoStrategy.atMost(1000).build() +strategy = DefaultVideoStrategy.atMost(1000).build(); // Ensures that minor and major dimension are at most the given values - scales down otherwise. -strategy = DefaultVideoStrategy.atMost(500, 1000).build() +strategy = DefaultVideoStrategy.atMost(500, 1000).build(); ``` In fact, all of these will simply call `new DefaultVideoStrategy.Builder(resizer)` with a special @@ -270,14 +275,14 @@ You can also group resizers through `MultiResizer`, which applies resizers in ch ```java // First scales down, then ensures size is at most 1000. Order matters! -Resizer resizer = new MultiResizer() -resizer.addResizer(new FractionResizer(0.5F)) -resizer.addResizer(new AtMostResizer(1000)) +Resizer resizer = new MultiResizer(); +resizer.addResizer(new FractionResizer(0.5F)); +resizer.addResizer(new AtMostResizer(1000)); // First makes it 16:9, then ensures size is at most 1000. Order matters! -Resizer resizer = new MultiResizer() -resizer.addResizer(new AspectRatioResizer(16F / 9F)) -resizer.addResizer(new AtMostResizer(1000)) +Resizer resizer = new MultiResizer(); +resizer.addResizer(new AspectRatioResizer(16F / 9F)); +resizer.addResizer(new AtMostResizer(1000)); ``` This option is already available through the DefaultVideoStrategy builder, so you can do: @@ -287,7 +292,7 @@ DefaultVideoStrategy strategy = new DefaultVideoStrategy.Builder() .addResizer(new AspectRatioResizer(16F / 9F)) .addResizer(new FractionResizer(0.5F)) .addResizer(new AtMostResizer(1000)) - .build() + .build(); ``` ### Other options @@ -300,10 +305,10 @@ DefaultVideoStrategy strategy = new DefaultVideoStrategy.Builder() .bitRate(DefaultVideoStrategy.BITRATE_UNKNOWN) // tries to estimate .frameRate(frameRate) // will be capped to the input frameRate .iFrameInterval(interval) // interval between I-frames in seconds - .build() + .build(); ``` -## Other Options +## Advanced Options #### Video rotation @@ -316,6 +321,70 @@ Transcoder.into(filePath) // ... ``` +#### Time interpolation + +We offer APIs to change the timestamp of each video and audio frame. You can pass a `TimeInterpolator` +to the transcoder builder to be able to receive the frame timestamp as input, and return a new one +as output. + +```java +Transcoder.into(filePath) + .setTimeInterpolator(timeInterpolator) + // ... +``` + +As an example, this is the implementation of the default interpolator, called `DefaultTimeInterpolator`, +that will just return the input time unchanged: + +```java +@Override +public long interpolate(@NonNull TrackType type, long time) { + // Receive input time in microseconds and return a possibly different one. + return time; +} +``` + +It should be obvious that returning invalid times can make the process crash at any point, or at least +the transcoding operation fail. + +#### Video speed + +We also offer a special time interpolator called `SpeedTimeInterpolator` that accepts a `float` parameter +and will modify the video speed. + +- A speed factor equal to 1 will leave speed unchanged +- A speed factor < 1 will slow the video down +- A speed factor > 1 will accelerate the video + +This interpolator can be set using `setTimeInterpolator(TimeInterpolator)`, or, as a shorthand, +using `setSpeed(float)`: + +```java +Transcoder.into(filePath) + .setSpeed(0.5F) // 0.5x + .setSpeed(1F) // Unchanged + .setSpeed(2F) // Twice as fast + // ... +``` + +#### Audio stretching + +When a time interpolator alters the frames and samples timestamps, you can either remove audio or +stretch the audio samples to the new length. This is done through the `AudioStretcher` interface: + +```java +Transcoder.into(filePath) + .setAudioStretcher(audioStretcher) + // ... +``` + +The default audio stretcher, `DefaultAudioStretcher`, will: + +- When we need to shrink a group of samples, cut the last ones +- When we need to stretch a group of samples, insert noise samples in between + +Please take a look at the implementation and read class documentation. + ## Compatibility As stated pretty much everywhere, **not all codecs/devices/manufacturers support all sizes/options**. @@ -323,7 +392,7 @@ This is a complex issue which is especially important for video strategies, as a to a transcoding error or corrupted file. Android platform specifies requirements for manufacturers through the [CTS (Compatibility test suite)](https://source.android.com/compatibility/cts). -Only a few codecs and sizes are strictly required to work. +Only a few codecs and sizes are **strictly** required to work. We collect common presets in the `DefaultVideoStrategies` class: diff --git a/demo/src/main/java/com/otaliastudios/transcoder/demo/TranscoderActivity.java b/demo/src/main/java/com/otaliastudios/transcoder/demo/TranscoderActivity.java index 1ec9f17d..8a76210a 100644 --- a/demo/src/main/java/com/otaliastudios/transcoder/demo/TranscoderActivity.java +++ b/demo/src/main/java/com/otaliastudios/transcoder/demo/TranscoderActivity.java @@ -11,13 +11,16 @@ import com.otaliastudios.transcoder.Transcoder; import com.otaliastudios.transcoder.TranscoderListener; +import com.otaliastudios.transcoder.engine.TrackStatus; import com.otaliastudios.transcoder.internal.Logger; import com.otaliastudios.transcoder.strategy.DefaultAudioStrategy; import com.otaliastudios.transcoder.strategy.DefaultVideoStrategy; import com.otaliastudios.transcoder.strategy.OutputStrategy; +import com.otaliastudios.transcoder.strategy.RemoveTrackStrategy; import com.otaliastudios.transcoder.strategy.size.AspectRatioResizer; import com.otaliastudios.transcoder.strategy.size.FractionResizer; import com.otaliastudios.transcoder.strategy.size.PassThroughResizer; +import com.otaliastudios.transcoder.validator.DefaultValidator; import java.io.File; import java.io.IOException; @@ -44,6 +47,8 @@ public class TranscoderActivity extends AppCompatActivity implements private RadioGroup mVideoResolutionGroup; private RadioGroup mVideoAspectGroup; private RadioGroup mVideoRotationGroup; + private RadioGroup mSpeedGroup; + private ProgressBar mProgressView; private TextView mButtonView; @@ -79,6 +84,8 @@ protected void onCreate(Bundle savedInstanceState) { mVideoResolutionGroup = findViewById(R.id.resolution); mVideoAspectGroup = findViewById(R.id.aspect); mVideoRotationGroup = findViewById(R.id.rotation); + mSpeedGroup = findViewById(R.id.speed); + mAudioChannelsGroup.setOnCheckedChangeListener(this); mVideoFramesGroup.setOnCheckedChangeListener(this); mVideoResolutionGroup.setOnCheckedChangeListener(this); @@ -166,6 +173,13 @@ private void transcode() { default: rotation = 0; } + float speed; + switch (mSpeedGroup.getCheckedRadioButtonId()) { + case R.id.speed_05x: speed = 0.5F; break; + case R.id.speed_2x: speed = 2F; break; + default: speed = 1F; + } + // Launch the transcoding operation. mTranscodeStartTime = SystemClock.uptimeMillis(); setIsTranscoding(true); @@ -175,6 +189,7 @@ private void transcode() { .setAudioOutputStrategy(mTranscodeAudioStrategy) .setVideoOutputStrategy(mTranscodeVideoStrategy) .setRotation(rotation) + .setSpeed(speed) .transcode(); } diff --git a/demo/src/main/res/layout/activity_transcoder.xml b/demo/src/main/res/layout/activity_transcoder.xml index 154c800e..3ddd3312 100644 --- a/demo/src/main/res/layout/activity_transcoder.xml +++ b/demo/src/main/res/layout/activity_transcoder.xml @@ -212,6 +212,43 @@ android:layout_height="wrap_content" /> + + + + + + + + + 0 && loopCount % PROGRESS_INTERVAL_STEPS == 0) { double videoProgress = getTranscoderProgress(videoTranscoder, mTracks.status(TrackType.VIDEO)); double audioProgress = getTranscoderProgress(audioTranscoder, mTracks.status(TrackType.AUDIO)); + LOG.i("progress - video:" + videoProgress + " audio:" + audioProgress); double progress = (videoProgress + audioProgress) / getTranscodersCount(); mProgress = progress; if (mProgressCallback != null) mProgressCallback.onProgress(progress); @@ -277,7 +278,7 @@ private void runPipelines() throws InterruptedException { private double getTranscoderProgress(@NonNull TrackTranscoder transcoder, @NonNull TrackStatus status) { if (!status.isTranscoding()) return 0.0; if (transcoder.isFinished()) return 1.0; - return Math.min(1.0, (double) transcoder.getLastWrittenPresentationTime() / mDurationUs); + return Math.min(1.0, (double) transcoder.getLastPresentationTime() / mDurationUs); } private int getTranscodersCount() { diff --git a/lib/src/main/java/com/otaliastudios/transcoder/internal/MediaCodecBuffers.java b/lib/src/main/java/com/otaliastudios/transcoder/internal/MediaCodecBuffers.java index b7315cc0..86d60ead 100644 --- a/lib/src/main/java/com/otaliastudios/transcoder/internal/MediaCodecBuffers.java +++ b/lib/src/main/java/com/otaliastudios/transcoder/internal/MediaCodecBuffers.java @@ -36,7 +36,9 @@ public ByteBuffer getInputBuffer(final int index) { //noinspection ConstantConditions return mMediaCodec.getInputBuffer(index); } - return mInputBuffers[index]; + ByteBuffer result = mInputBuffers[index]; + result.clear(); + return result; } @NonNull diff --git a/lib/src/main/java/com/otaliastudios/transcoder/remix/AudioRemixer.java b/lib/src/main/java/com/otaliastudios/transcoder/remix/AudioRemixer.java index ca7767a7..ce80f45e 100644 --- a/lib/src/main/java/com/otaliastudios/transcoder/remix/AudioRemixer.java +++ b/lib/src/main/java/com/otaliastudios/transcoder/remix/AudioRemixer.java @@ -2,6 +2,7 @@ import androidx.annotation.NonNull; +import java.nio.Buffer; import java.nio.ShortBuffer; /** @@ -11,7 +12,23 @@ */ public interface AudioRemixer { - void remix(@NonNull final ShortBuffer inSBuff, @NonNull final ShortBuffer outSBuff); + /** + * Remixes input audio from input buffer into the output buffer. + * The output buffer is guaranteed to have a {@link Buffer#remaining()} size that is + * consistent with {@link #getRemixedSize(int)}. + * + * @param inputBuffer the input buffer + * @param outputBuffer the output buffer + */ + void remix(@NonNull final ShortBuffer inputBuffer, @NonNull final ShortBuffer outputBuffer); + + /** + * Returns the output size (in shorts) needed to process an input buffer of the + * given size (in shorts). + * @param inputSize input size in shorts + * @return output size in shorts + */ + int getRemixedSize(int inputSize); AudioRemixer DOWNMIX = new DownMixAudioRemixer(); diff --git a/lib/src/main/java/com/otaliastudios/transcoder/remix/DownMixAudioRemixer.java b/lib/src/main/java/com/otaliastudios/transcoder/remix/DownMixAudioRemixer.java index 03bb4692..6385fdd8 100644 --- a/lib/src/main/java/com/otaliastudios/transcoder/remix/DownMixAudioRemixer.java +++ b/lib/src/main/java/com/otaliastudios/transcoder/remix/DownMixAudioRemixer.java @@ -13,19 +13,19 @@ public class DownMixAudioRemixer implements AudioRemixer { private static final int UNSIGNED_SHORT_MAX = 65535; @Override - public void remix(@NonNull final ShortBuffer inSBuff, @NonNull final ShortBuffer outSBuff) { + public void remix(@NonNull final ShortBuffer inputBuffer, @NonNull final ShortBuffer outputBuffer) { // Down-mix stereo to mono // Viktor Toth's algorithm - // See: http://www.vttoth.com/CMS/index.php/technical-notes/68 // http://stackoverflow.com/a/25102339 - final int inRemaining = inSBuff.remaining() / 2; - final int outSpace = outSBuff.remaining(); + final int inRemaining = inputBuffer.remaining() / 2; + final int outSpace = outputBuffer.remaining(); final int samplesToBeProcessed = Math.min(inRemaining, outSpace); for (int i = 0; i < samplesToBeProcessed; ++i) { // Convert to unsigned - final int a = inSBuff.get() + SIGNED_SHORT_LIMIT; - final int b = inSBuff.get() + SIGNED_SHORT_LIMIT; + final int a = inputBuffer.get() + SIGNED_SHORT_LIMIT; + final int b = inputBuffer.get() + SIGNED_SHORT_LIMIT; int m; // Pick the equation if ((a < SIGNED_SHORT_LIMIT) || (b < SIGNED_SHORT_LIMIT)) { @@ -38,7 +38,12 @@ public void remix(@NonNull final ShortBuffer inSBuff, @NonNull final ShortBuffer } // Convert output back to signed short if (m == UNSIGNED_SHORT_MAX + 1) m = UNSIGNED_SHORT_MAX; - outSBuff.put((short) (m - SIGNED_SHORT_LIMIT)); + outputBuffer.put((short) (m - SIGNED_SHORT_LIMIT)); } } + + @Override + public int getRemixedSize(int inputSize) { + return inputSize / 2; + } } diff --git a/lib/src/main/java/com/otaliastudios/transcoder/remix/PassThroughAudioRemixer.java b/lib/src/main/java/com/otaliastudios/transcoder/remix/PassThroughAudioRemixer.java index 2d43c052..717d7390 100644 --- a/lib/src/main/java/com/otaliastudios/transcoder/remix/PassThroughAudioRemixer.java +++ b/lib/src/main/java/com/otaliastudios/transcoder/remix/PassThroughAudioRemixer.java @@ -10,8 +10,12 @@ public class PassThroughAudioRemixer implements AudioRemixer { @Override - public void remix(@NonNull final ShortBuffer inSBuff, @NonNull final ShortBuffer outSBuff) { - // Passthrough - outSBuff.put(inSBuff); + public void remix(@NonNull final ShortBuffer inputBuffer, @NonNull final ShortBuffer outputBuffer) { + outputBuffer.put(inputBuffer); + } + + @Override + public int getRemixedSize(int inputSize) { + return inputSize; } } diff --git a/lib/src/main/java/com/otaliastudios/transcoder/remix/UpMixAudioRemixer.java b/lib/src/main/java/com/otaliastudios/transcoder/remix/UpMixAudioRemixer.java index 3baf6737..a0e27ff0 100644 --- a/lib/src/main/java/com/otaliastudios/transcoder/remix/UpMixAudioRemixer.java +++ b/lib/src/main/java/com/otaliastudios/transcoder/remix/UpMixAudioRemixer.java @@ -10,16 +10,21 @@ public class UpMixAudioRemixer implements AudioRemixer { @Override - public void remix(@NonNull final ShortBuffer inSBuff, @NonNull final ShortBuffer outSBuff) { + public void remix(@NonNull final ShortBuffer inputBuffer, @NonNull final ShortBuffer outputBuffer) { // Up-mix mono to stereo - final int inRemaining = inSBuff.remaining(); - final int outSpace = outSBuff.remaining() / 2; + final int inRemaining = inputBuffer.remaining(); + final int outSpace = outputBuffer.remaining() / 2; final int samplesToBeProcessed = Math.min(inRemaining, outSpace); for (int i = 0; i < samplesToBeProcessed; ++i) { - final short inSample = inSBuff.get(); - outSBuff.put(inSample); - outSBuff.put(inSample); + final short inSample = inputBuffer.get(); + outputBuffer.put(inSample); + outputBuffer.put(inSample); } } + + @Override + public int getRemixedSize(int inputSize) { + return inputSize * 2; + } } diff --git a/lib/src/main/java/com/otaliastudios/transcoder/stretch/AudioStretcher.java b/lib/src/main/java/com/otaliastudios/transcoder/stretch/AudioStretcher.java new file mode 100644 index 00000000..e35c4f26 --- /dev/null +++ b/lib/src/main/java/com/otaliastudios/transcoder/stretch/AudioStretcher.java @@ -0,0 +1,50 @@ +package com.otaliastudios.transcoder.stretch; + +import androidx.annotation.NonNull; + +import java.nio.Buffer; +import java.nio.ShortBuffer; +import java.util.Random; + +/** + * An AudioStretcher will change audio samples duration, in response to a + * {@link com.otaliastudios.transcoder.time.TimeInterpolator} that altered the sample timestamp. + * + * This can mean either shrink the sample (in case of video speed up) or elongate it (in case of + * video slow down) so that it matches the output size. + */ +public interface AudioStretcher { + + /** + * Stretches the input into the output, based on the {@link Buffer#remaining()} value of both. + * At the end of this method, the {@link Buffer#position()} of both should be equal to their + * respective {@link Buffer#limit()}. + * + * And of course, both {@link Buffer#limit()}s should remain unchanged. + * + * @param input input buffer + * @param output output buffer + * @param channels audio channels + */ + void stretch(@NonNull ShortBuffer input, @NonNull ShortBuffer output, int channels); + + AudioStretcher PASSTHROUGH = new PassThroughAudioStretcher(); + + AudioStretcher CUT = new CutAudioStretcher(); + + AudioStretcher INSERT = new InsertAudioStretcher(); + + + AudioStretcher CUT_OR_INSERT = new AudioStretcher() { + @Override + public void stretch(@NonNull ShortBuffer input, @NonNull ShortBuffer output, int channels) { + if (input.remaining() < output.remaining()) { + INSERT.stretch(input, output, channels); + } else if (input.remaining() > output.remaining()) { + CUT.stretch(input, output, channels); + } else { + PASSTHROUGH.stretch(input, output, channels); + } + } + }; +} diff --git a/lib/src/main/java/com/otaliastudios/transcoder/stretch/CutAudioStretcher.java b/lib/src/main/java/com/otaliastudios/transcoder/stretch/CutAudioStretcher.java new file mode 100644 index 00000000..c2ea24f8 --- /dev/null +++ b/lib/src/main/java/com/otaliastudios/transcoder/stretch/CutAudioStretcher.java @@ -0,0 +1,24 @@ +package com.otaliastudios.transcoder.stretch; + +import androidx.annotation.NonNull; + +import java.nio.ShortBuffer; + +/** + * A {@link AudioStretcher} meant to be used when output size is smaller than the input. + * Cutting the latest samples is a way to go that does not modify the audio pitch. + */ +public class CutAudioStretcher implements AudioStretcher { + + @Override + public void stretch(@NonNull ShortBuffer input, @NonNull ShortBuffer output, int channels) { + if (input.remaining() < output.remaining()) { + throw new IllegalArgumentException("Illegal use of CutAudioStretcher"); + } + int exceeding = input.remaining() - output.remaining(); + input.limit(input.limit() - exceeding); // Make remaining() the same for both + output.put(input); // Safely bulk-put + input.limit(input.limit() + exceeding); // Restore + input.position(input.limit()); // Make as if we have read it all + } +} diff --git a/lib/src/main/java/com/otaliastudios/transcoder/stretch/DefaultAudioStretcher.java b/lib/src/main/java/com/otaliastudios/transcoder/stretch/DefaultAudioStretcher.java new file mode 100644 index 00000000..b83726ab --- /dev/null +++ b/lib/src/main/java/com/otaliastudios/transcoder/stretch/DefaultAudioStretcher.java @@ -0,0 +1,23 @@ +package com.otaliastudios.transcoder.stretch; + +import androidx.annotation.NonNull; + +import java.nio.ShortBuffer; + +/** + * An {@link AudioStretcher} that delegates to appropriate classes + * based on input and output size. + */ +public class DefaultAudioStretcher implements AudioStretcher { + + @Override + public void stretch(@NonNull ShortBuffer input, @NonNull ShortBuffer output, int channels) { + if (input.remaining() < output.remaining()) { + INSERT.stretch(input, output, channels); + } else if (input.remaining() > output.remaining()) { + CUT.stretch(input, output, channels); + } else { + PASSTHROUGH.stretch(input, output, channels); + } + } +} diff --git a/lib/src/main/java/com/otaliastudios/transcoder/stretch/InsertAudioStretcher.java b/lib/src/main/java/com/otaliastudios/transcoder/stretch/InsertAudioStretcher.java new file mode 100644 index 00000000..7c104cd5 --- /dev/null +++ b/lib/src/main/java/com/otaliastudios/transcoder/stretch/InsertAudioStretcher.java @@ -0,0 +1,55 @@ +package com.otaliastudios.transcoder.stretch; + +import androidx.annotation.NonNull; + +import java.nio.ShortBuffer; +import java.util.Random; + +/** + * A {@link AudioStretcher} meant to be used when output size is bigger than the input. + * It will insert noise samples to fill the gaps, at regular intervals. + * This modifies the audio pitch of course. + */ +public class InsertAudioStretcher implements AudioStretcher { + + private final static Random NOISE = new Random(); + + private static short noise() { + return (short) NOISE.nextInt(300); + } + + private static float ratio(int remaining, int all) { + return (float) remaining / all; + } + + @Override + public void stretch(@NonNull ShortBuffer input, @NonNull ShortBuffer output, int channels) { + if (input.remaining() >= output.remaining()) { + throw new IllegalArgumentException("Illegal use of AudioStretcher.INSERT"); + } + if (channels != 1 && channels != 2) { + throw new IllegalArgumentException("Illegal use of AudioStretcher.INSERT. Channels:" + channels); + } + final int inputSamples = input.remaining() / channels; + final int fakeSamples = (int) Math.floor((double) (output.remaining() - input.remaining()) / channels); + int remainingInputSamples = inputSamples; + int remainingFakeSamples = fakeSamples; + float remainingInputSamplesRatio = ratio(remainingInputSamples, inputSamples); + float remainingFakeSamplesRatio = ratio(remainingFakeSamples, fakeSamples); + while (remainingInputSamples > 0 && remainingFakeSamples > 0) { + // Will this be an input sample or a fake sample? + // Choose the one with the bigger ratio. + if (remainingInputSamplesRatio >= remainingFakeSamplesRatio) { + output.put(input.get()); + if (channels == 2) output.put(input.get()); + remainingInputSamples--; + remainingInputSamplesRatio = ratio(remainingInputSamples, inputSamples); + } else { + output.put(noise()); + if (channels == 2) output.put(noise()); + remainingFakeSamples--; + remainingFakeSamplesRatio = ratio(remainingFakeSamples, inputSamples); + } + } + } +} diff --git a/lib/src/main/java/com/otaliastudios/transcoder/stretch/PassThroughAudioStretcher.java b/lib/src/main/java/com/otaliastudios/transcoder/stretch/PassThroughAudioStretcher.java new file mode 100644 index 00000000..14768c8f --- /dev/null +++ b/lib/src/main/java/com/otaliastudios/transcoder/stretch/PassThroughAudioStretcher.java @@ -0,0 +1,19 @@ +package com.otaliastudios.transcoder.stretch; + +import androidx.annotation.NonNull; + +import java.nio.ShortBuffer; + +/** + * A no-op {@link AudioStretcher} that copies input into output. + */ +public class PassThroughAudioStretcher implements AudioStretcher { + + @Override + public void stretch(@NonNull ShortBuffer input, @NonNull ShortBuffer output, int channels) { + if (input.remaining() > output.remaining()) { + throw new IllegalArgumentException("Illegal use of PassThroughAudioStretcher"); + } + output.put(input); + } +} diff --git a/lib/src/main/java/com/otaliastudios/transcoder/time/DefaultTimeInterpolator.java b/lib/src/main/java/com/otaliastudios/transcoder/time/DefaultTimeInterpolator.java new file mode 100644 index 00000000..d4936afc --- /dev/null +++ b/lib/src/main/java/com/otaliastudios/transcoder/time/DefaultTimeInterpolator.java @@ -0,0 +1,17 @@ +package com.otaliastudios.transcoder.time; + +import androidx.annotation.NonNull; + +import com.otaliastudios.transcoder.engine.TrackType; + +/** + * A {@link TimeInterpolator} that does no time interpolation or correction - + * it just returns the input time. + */ +public class DefaultTimeInterpolator implements TimeInterpolator { + + @Override + public long interpolate(@NonNull TrackType type, long time) { + return time; + } +} diff --git a/lib/src/main/java/com/otaliastudios/transcoder/time/SpeedTimeInterpolator.java b/lib/src/main/java/com/otaliastudios/transcoder/time/SpeedTimeInterpolator.java new file mode 100644 index 00000000..d5853195 --- /dev/null +++ b/lib/src/main/java/com/otaliastudios/transcoder/time/SpeedTimeInterpolator.java @@ -0,0 +1,71 @@ +package com.otaliastudios.transcoder.time; + +import android.util.Log; + +import androidx.annotation.NonNull; + +import com.otaliastudios.transcoder.engine.TrackType; +import com.otaliastudios.transcoder.internal.Logger; + +import java.util.HashMap; +import java.util.Map; + +/** + * A {@link TimeInterpolator} that modifies the playback speed by the given + * float factor. A factor less than 1 will slow down, while a bigger factor will + * accelerate. + */ +public class SpeedTimeInterpolator implements TimeInterpolator { + + private final static String TAG = SpeedTimeInterpolator.class.getSimpleName(); + private final static Logger LOG = new Logger(TAG); + + private double mFactor; + private final Map mTrackData = new HashMap<>(); + + /** + * Creates a new speed interpolator for the given factor. + * Throws if factor is less than 0 or equal to 0. + * @param factor a factor + */ + public SpeedTimeInterpolator(float factor) { + if (factor <= 0) { + throw new IllegalArgumentException("Invalid speed factor: " + factor); + } + mFactor = factor; + } + + /** + * Returns the factor passed to the constructor. + * @return the factor + */ + @SuppressWarnings("unused") + public float getFactor() { + return (float) mFactor; + } + + @Override + public long interpolate(@NonNull TrackType type, long time) { + if (!mTrackData.containsKey(type)) { + mTrackData.put(type, new TrackData()); + } + TrackData data = mTrackData.get(type); + //noinspection ConstantConditions + if (data.lastRealTime == Long.MIN_VALUE) { + data.lastRealTime = time; + data.lastCorrectedTime = time; + } else { + long realDelta = time - data.lastRealTime; + long correctedDelta = (long) ((double) realDelta / mFactor); + data.lastRealTime = time; + data.lastCorrectedTime += correctedDelta; + } + LOG.i("Track:" + type + " inputTime:" + time + " outputTime:" + data.lastCorrectedTime); + return data.lastCorrectedTime; + } + + private static class TrackData { + private long lastRealTime = Long.MIN_VALUE; + private long lastCorrectedTime = Long.MIN_VALUE; + } +} diff --git a/lib/src/main/java/com/otaliastudios/transcoder/time/TimeInterpolator.java b/lib/src/main/java/com/otaliastudios/transcoder/time/TimeInterpolator.java new file mode 100644 index 00000000..515b4471 --- /dev/null +++ b/lib/src/main/java/com/otaliastudios/transcoder/time/TimeInterpolator.java @@ -0,0 +1,21 @@ +package com.otaliastudios.transcoder.time; + +import androidx.annotation.NonNull; + +import com.otaliastudios.transcoder.engine.TrackType; + +/** + * An interface to redefine the time between video or audio frames. + */ +public interface TimeInterpolator { + + /** + * Given the track type (audio or video) and the frame timestamp in microseconds, + * should return the corrected timestamp. + * + * @param type track type + * @param time frame timestamp in microseconds + * @return the new frame timestamp + */ + long interpolate(@NonNull TrackType type, long time); +} diff --git a/lib/src/main/java/com/otaliastudios/transcoder/transcode/AudioTrackTranscoder.java b/lib/src/main/java/com/otaliastudios/transcoder/transcode/AudioTrackTranscoder.java index ac9aaeed..740bf3ea 100644 --- a/lib/src/main/java/com/otaliastudios/transcoder/transcode/AudioTrackTranscoder.java +++ b/lib/src/main/java/com/otaliastudios/transcoder/transcode/AudioTrackTranscoder.java @@ -5,205 +5,59 @@ import android.media.MediaFormat; import androidx.annotation.NonNull; -import androidx.annotation.Nullable; import com.otaliastudios.transcoder.engine.TrackType; import com.otaliastudios.transcoder.engine.TranscoderMuxer; import com.otaliastudios.transcoder.internal.MediaCodecBuffers; -import com.otaliastudios.transcoder.transcode.internal.AudioChannel; +import com.otaliastudios.transcoder.stretch.AudioStretcher; +import com.otaliastudios.transcoder.time.TimeInterpolator; +import com.otaliastudios.transcoder.transcode.internal.AudioEngine; -import java.io.IOException; +import java.nio.ByteBuffer; -public class AudioTrackTranscoder implements TrackTranscoder { +public class AudioTrackTranscoder extends BaseTrackTranscoder { - private static final TrackType SAMPLE_TYPE = TrackType.AUDIO; - - private static final int DRAIN_STATE_NONE = 0; - private static final int DRAIN_STATE_SHOULD_RETRY_IMMEDIATELY = 1; - private static final int DRAIN_STATE_CONSUMED = 2; - - private final MediaExtractor mExtractor; - private final TranscoderMuxer mMuxer; - private long mWrittenPresentationTimeUs; - - private final int mTrackIndex; - - private final MediaCodec.BufferInfo mBufferInfo = new MediaCodec.BufferInfo(); - private MediaCodec mDecoder; - private MediaCodec mEncoder; - private MediaFormat mActualOutputFormat; - - private MediaCodecBuffers mDecoderBuffers; - private MediaCodecBuffers mEncoderBuffers; - - private boolean mIsExtractorEOS; - private boolean mIsDecoderEOS; - private boolean mIsEncoderEOS; - private boolean mDecoderStarted; - private boolean mEncoderStarted; - - private AudioChannel mAudioChannel; + private TimeInterpolator mTimeInterpolator; + private AudioStretcher mAudioStretcher; + private AudioEngine mAudioEngine; + private MediaCodec mEncoder; // to create the channel + private MediaFormat mEncoderOutputFormat; // to create the channel public AudioTrackTranscoder(@NonNull MediaExtractor extractor, + @NonNull TranscoderMuxer muxer, int trackIndex, - @NonNull TranscoderMuxer muxer) { - mExtractor = extractor; - mTrackIndex = trackIndex; - mMuxer = muxer; + @NonNull TimeInterpolator timeInterpolator, + @NonNull AudioStretcher audioStretcher) { + super(extractor, muxer, TrackType.AUDIO, trackIndex); + mTimeInterpolator = timeInterpolator; + mAudioStretcher = audioStretcher; } @Override - public void setUp(@NonNull MediaFormat desiredOutputFormat) { - mExtractor.selectTrack(mTrackIndex); - try { - mEncoder = MediaCodec.createEncoderByType(desiredOutputFormat.getString(MediaFormat.KEY_MIME)); - } catch (IOException e) { - throw new IllegalStateException(e); - } - mEncoder.configure(desiredOutputFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE); - mEncoder.start(); - mEncoderStarted = true; - mEncoderBuffers = new MediaCodecBuffers(mEncoder); - - final MediaFormat inputFormat = mExtractor.getTrackFormat(mTrackIndex); - try { - mDecoder = MediaCodec.createDecoderByType(inputFormat.getString(MediaFormat.KEY_MIME)); - } catch (IOException e) { - throw new IllegalStateException(e); - } - mDecoder.configure(inputFormat, null, null, 0); - mDecoder.start(); - mDecoderStarted = true; - mDecoderBuffers = new MediaCodecBuffers(mDecoder); - - mAudioChannel = new AudioChannel(mDecoder, mEncoder, desiredOutputFormat); - } - - @Override - public boolean stepPipeline() { - boolean busy = false; - - int status; - while (drainEncoder(0) != DRAIN_STATE_NONE) busy = true; - do { - status = drainDecoder(0); - if (status != DRAIN_STATE_NONE) busy = true; - // NOTE: not repeating to keep from deadlock when encoder is full. - } while (status == DRAIN_STATE_SHOULD_RETRY_IMMEDIATELY); - - while (mAudioChannel.feedEncoder(0)) busy = true; - while (drainExtractor(0) != DRAIN_STATE_NONE) busy = true; - - return busy; - } - - @SuppressWarnings("SameParameterValue") - private int drainExtractor(long timeoutUs) { - if (mIsExtractorEOS) return DRAIN_STATE_NONE; - int trackIndex = mExtractor.getSampleTrackIndex(); - if (trackIndex >= 0 && trackIndex != mTrackIndex) { - return DRAIN_STATE_NONE; - } - - final int result = mDecoder.dequeueInputBuffer(timeoutUs); - if (result < 0) return DRAIN_STATE_NONE; - if (trackIndex < 0) { - mIsExtractorEOS = true; - mDecoder.queueInputBuffer(result, 0, 0, 0, MediaCodec.BUFFER_FLAG_END_OF_STREAM); - return DRAIN_STATE_NONE; - } - - final int sampleSize = mExtractor.readSampleData(mDecoderBuffers.getInputBuffer(result), 0); - final boolean isKeyFrame = (mExtractor.getSampleFlags() & MediaExtractor.SAMPLE_FLAG_SYNC) != 0; - mDecoder.queueInputBuffer(result, 0, sampleSize, mExtractor.getSampleTime(), isKeyFrame ? MediaCodec.BUFFER_FLAG_SYNC_FRAME : 0); - mExtractor.advance(); - return DRAIN_STATE_CONSUMED; - } - - @SuppressWarnings("SameParameterValue") - private int drainDecoder(long timeoutUs) { - if (mIsDecoderEOS) return DRAIN_STATE_NONE; - - int result = mDecoder.dequeueOutputBuffer(mBufferInfo, timeoutUs); - switch (result) { - case MediaCodec.INFO_TRY_AGAIN_LATER: - return DRAIN_STATE_NONE; - case MediaCodec.INFO_OUTPUT_FORMAT_CHANGED: - mAudioChannel.setActualDecodedFormat(mDecoder.getOutputFormat()); - case MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED: - return DRAIN_STATE_SHOULD_RETRY_IMMEDIATELY; - } - - if ((mBufferInfo.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) { - mIsDecoderEOS = true; - mAudioChannel.drainDecoderBufferAndQueue(AudioChannel.BUFFER_INDEX_END_OF_STREAM, 0); - } else if (mBufferInfo.size > 0) { - mAudioChannel.drainDecoderBufferAndQueue(result, mBufferInfo.presentationTimeUs); - } - - return DRAIN_STATE_CONSUMED; - } - - @SuppressWarnings("SameParameterValue") - private int drainEncoder(long timeoutUs) { - if (mIsEncoderEOS) return DRAIN_STATE_NONE; - - int result = mEncoder.dequeueOutputBuffer(mBufferInfo, timeoutUs); - switch (result) { - case MediaCodec.INFO_TRY_AGAIN_LATER: - return DRAIN_STATE_NONE; - case MediaCodec.INFO_OUTPUT_FORMAT_CHANGED: - if (mActualOutputFormat != null) { - throw new RuntimeException("Audio output format changed twice."); - } - mActualOutputFormat = mEncoder.getOutputFormat(); - mMuxer.setOutputFormat(SAMPLE_TYPE, mActualOutputFormat); - return DRAIN_STATE_SHOULD_RETRY_IMMEDIATELY; - case MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED: - mEncoderBuffers = new MediaCodecBuffers(mEncoder); - return DRAIN_STATE_SHOULD_RETRY_IMMEDIATELY; - } - - if (mActualOutputFormat == null) { - throw new RuntimeException("Could not determine actual output format."); - } - - if ((mBufferInfo.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) { - mIsEncoderEOS = true; - mBufferInfo.set(0, 0, 0, mBufferInfo.flags); - } - if ((mBufferInfo.flags & MediaCodec.BUFFER_FLAG_CODEC_CONFIG) != 0) { - // SPS or PPS, which should be passed by MediaFormat. - mEncoder.releaseOutputBuffer(result, false); - return DRAIN_STATE_SHOULD_RETRY_IMMEDIATELY; - } - mMuxer.write(SAMPLE_TYPE, mEncoderBuffers.getOutputBuffer(result), mBufferInfo); - mWrittenPresentationTimeUs = mBufferInfo.presentationTimeUs; - mEncoder.releaseOutputBuffer(result, false); - return DRAIN_STATE_CONSUMED; + protected void onCodecsStarted(@NonNull MediaFormat inputFormat, @NonNull MediaFormat outputFormat, @NonNull MediaCodec decoder, @NonNull MediaCodec encoder) { + super.onCodecsStarted(inputFormat, outputFormat, decoder, encoder); + mEncoder = encoder; + mEncoderOutputFormat = outputFormat; } @Override - public long getLastWrittenPresentationTime() { - return mWrittenPresentationTimeUs; + protected boolean onFeedEncoder(@NonNull MediaCodec encoder, @NonNull MediaCodecBuffers encoderBuffers, long timeoutUs) { + if (mAudioEngine == null) return false; + return mAudioEngine.feedEncoder(encoderBuffers, timeoutUs); } @Override - public boolean isFinished() { - return mIsEncoderEOS; + protected void onDecoderOutputFormatChanged(@NonNull MediaCodec decoder, @NonNull MediaFormat format) { + super.onDecoderOutputFormatChanged(decoder, format); + mAudioEngine = new AudioEngine(decoder, format, mEncoder, mEncoderOutputFormat, mTimeInterpolator, mAudioStretcher); + mEncoder = null; + mEncoderOutputFormat = null; + mTimeInterpolator = null; + mAudioStretcher = null; } @Override - public void release() { - if (mDecoder != null) { - if (mDecoderStarted) mDecoder.stop(); - mDecoder.release(); - mDecoder = null; - } - if (mEncoder != null) { - if (mEncoderStarted) mEncoder.stop(); - mEncoder.release(); - mEncoder = null; - } + protected void onDrainDecoder(@NonNull MediaCodec decoder, int bufferIndex, @NonNull ByteBuffer bufferData, long presentationTimeUs, boolean endOfStream) { + mAudioEngine.drainDecoder(bufferIndex, bufferData, presentationTimeUs, endOfStream); } } diff --git a/lib/src/main/java/com/otaliastudios/transcoder/transcode/BaseTrackTranscoder.java b/lib/src/main/java/com/otaliastudios/transcoder/transcode/BaseTrackTranscoder.java new file mode 100644 index 00000000..e2850338 --- /dev/null +++ b/lib/src/main/java/com/otaliastudios/transcoder/transcode/BaseTrackTranscoder.java @@ -0,0 +1,318 @@ +package com.otaliastudios.transcoder.transcode; + +import android.media.MediaCodec; +import android.media.MediaExtractor; +import android.media.MediaFormat; + +import androidx.annotation.CallSuper; +import androidx.annotation.NonNull; + +import com.otaliastudios.transcoder.engine.TrackType; +import com.otaliastudios.transcoder.engine.TranscoderMuxer; +import com.otaliastudios.transcoder.internal.MediaCodecBuffers; + +import java.io.IOException; +import java.nio.ByteBuffer; + +/** + * A base implementation of {@link TrackTranscoder} that reads + * from {@link MediaExtractor} and does feeding and draining job. + */ +public abstract class BaseTrackTranscoder implements TrackTranscoder { + + private static final int DRAIN_STATE_NONE = 0; + private static final int DRAIN_STATE_SHOULD_RETRY_IMMEDIATELY = 1; + private static final int DRAIN_STATE_CONSUMED = 2; + + private final MediaExtractor mExtractor; + private final int mTrackIndex; + private final TranscoderMuxer mMuxer; + private final TrackType mTrackType; + + private long mLastPresentationTimeUs; + + private final MediaCodec.BufferInfo mBufferInfo = new MediaCodec.BufferInfo(); + private MediaCodec mDecoder; + private MediaCodec mEncoder; + private MediaCodecBuffers mDecoderBuffers; + private MediaCodecBuffers mEncoderBuffers; + private boolean mDecoderStarted; + private boolean mEncoderStarted; + private MediaFormat mActualOutputFormat; + + private boolean mIsDecoderEOS; + private boolean mIsEncoderEOS; + private boolean mIsExtractorEOS; + + @SuppressWarnings("WeakerAccess") + protected BaseTrackTranscoder(@NonNull MediaExtractor extractor, + @NonNull TranscoderMuxer muxer, + @NonNull TrackType trackType, + int trackIndex) { + mExtractor = extractor; + mTrackIndex = trackIndex; + mMuxer = muxer; + mTrackType = trackType; + } + + @Override + public final void setUp(@NonNull MediaFormat desiredOutputFormat) { + mExtractor.selectTrack(mTrackIndex); + try { + mEncoder = MediaCodec.createEncoderByType(desiredOutputFormat.getString(MediaFormat.KEY_MIME)); + } catch (IOException e) { + throw new IllegalStateException(e); + } + onConfigureEncoder(desiredOutputFormat, mEncoder); + onStartEncoder(desiredOutputFormat, mEncoder); + + final MediaFormat inputFormat = mExtractor.getTrackFormat(mTrackIndex); + try { + mDecoder = MediaCodec.createDecoderByType(inputFormat.getString(MediaFormat.KEY_MIME)); + } catch (IOException e) { + throw new IllegalStateException(e); + } + onConfigureDecoder(inputFormat, mDecoder); + onStartDecoder(inputFormat, mDecoder); + onCodecsStarted(inputFormat, desiredOutputFormat, mDecoder, mEncoder); + } + + /** + * Wraps the configure operation on the encoder. + * @param format output format + * @param encoder encoder + */ + @SuppressWarnings("WeakerAccess") + protected void onConfigureEncoder(@NonNull MediaFormat format, @NonNull MediaCodec encoder) { + encoder.configure(format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE); + } + + /** + * Wraps the start operation on the encoder. + * @param format output format + * @param encoder encoder + */ + @CallSuper + protected void onStartEncoder(@NonNull MediaFormat format, @NonNull MediaCodec encoder) { + encoder.start(); + mEncoderStarted = true; + mEncoderBuffers = new MediaCodecBuffers(encoder); + } + + /** + * Wraps the configure operation on the decoder. + * @param format input format + * @param decoder decoder + */ + protected void onConfigureDecoder(@NonNull MediaFormat format, @NonNull MediaCodec decoder) { + decoder.configure(format, null, null, 0); + } + + /** + * Wraps the start operation on the decoder. + * @param format input format + * @param decoder decoder + */ + @SuppressWarnings({"WeakerAccess", "unused"}) + @CallSuper + protected void onStartDecoder(@NonNull MediaFormat format, @NonNull MediaCodec decoder) { + decoder.start(); + mDecoderStarted = true; + mDecoderBuffers = new MediaCodecBuffers(decoder); + } + + /** + * Called when both codecs have been started with the given formats. + * @param inputFormat input format + * @param outputFormat output format + * @param decoder decoder + * @param encoder encoder + */ + protected void onCodecsStarted(@NonNull MediaFormat inputFormat, @NonNull MediaFormat outputFormat, + @NonNull MediaCodec decoder, @NonNull MediaCodec encoder) { + } + + @Override + public final long getLastPresentationTime() { + return mLastPresentationTimeUs; + } + + @Override + public final boolean isFinished() { + return mIsEncoderEOS; + } + + @Override + public void release() { + if (mDecoder != null) { + if (mDecoderStarted) { + mDecoder.stop(); + mDecoderStarted = false; + } + mDecoder.release(); + mDecoder = null; + } + if (mEncoder != null) { + if (mEncoderStarted) { + mEncoder.stop(); + mEncoderStarted = false; + } + mEncoder.release(); + mEncoder = null; + } + } + + @Override + public final boolean transcode() { + boolean busy = false; + int status; + while (drainEncoder(0) != DRAIN_STATE_NONE) busy = true; + do { + status = drainDecoder(0); + if (status != DRAIN_STATE_NONE) busy = true; + // NOTE: not repeating to keep from deadlock when encoder is full. + } while (status == DRAIN_STATE_SHOULD_RETRY_IMMEDIATELY); + + while (feedEncoder(0)) busy = true; + while (feedDecoder(0) != DRAIN_STATE_NONE) busy = true; + return busy; + } + + /** + * Called when the decoder has defined its actual output format. + * @param format format + */ + @CallSuper + protected void onDecoderOutputFormatChanged(@NonNull MediaCodec decoder, @NonNull MediaFormat format) {} + + /** + * Called when the encoder has defined its actual output format. + * @param format format + */ + @CallSuper + @SuppressWarnings("WeakerAccess") + protected void onEncoderOutputFormatChanged(@NonNull MediaCodec encoder, @NonNull MediaFormat format) { + if (mActualOutputFormat != null) { + throw new RuntimeException("Audio output format changed twice."); + } + mActualOutputFormat = format; + mMuxer.setOutputFormat(mTrackType, mActualOutputFormat); + } + + @SuppressWarnings("SameParameterValue") + private int feedDecoder(long timeoutUs) { + if (mIsExtractorEOS) return DRAIN_STATE_NONE; + int trackIndex = mExtractor.getSampleTrackIndex(); + if (trackIndex >= 0 && trackIndex != mTrackIndex) { + return DRAIN_STATE_NONE; + } + + final int result = mDecoder.dequeueInputBuffer(timeoutUs); + if (result < 0) return DRAIN_STATE_NONE; + if (trackIndex < 0) { + mIsExtractorEOS = true; + mDecoder.queueInputBuffer(result, 0, 0, 0, MediaCodec.BUFFER_FLAG_END_OF_STREAM); + return DRAIN_STATE_NONE; + } + + final int sampleSize = mExtractor.readSampleData(mDecoderBuffers.getInputBuffer(result), 0); + final boolean isKeyFrame = (mExtractor.getSampleFlags() & MediaExtractor.SAMPLE_FLAG_SYNC) != 0; + mDecoder.queueInputBuffer(result, 0, sampleSize, mExtractor.getSampleTime(), isKeyFrame ? MediaCodec.BUFFER_FLAG_SYNC_FRAME : 0); + mExtractor.advance(); + return DRAIN_STATE_CONSUMED; + } + + @SuppressWarnings("SameParameterValue") + private boolean feedEncoder(long timeoutUs) { + return onFeedEncoder(mEncoder, mEncoderBuffers, timeoutUs); + } + + @SuppressWarnings("SameParameterValue") + private int drainDecoder(long timeoutUs) { + if (mIsDecoderEOS) return DRAIN_STATE_NONE; + int result = mDecoder.dequeueOutputBuffer(mBufferInfo, timeoutUs); + switch (result) { + case MediaCodec.INFO_TRY_AGAIN_LATER: + return DRAIN_STATE_NONE; + case MediaCodec.INFO_OUTPUT_FORMAT_CHANGED: + onDecoderOutputFormatChanged(mDecoder, mDecoder.getOutputFormat()); + return DRAIN_STATE_SHOULD_RETRY_IMMEDIATELY; + case MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED: + return DRAIN_STATE_SHOULD_RETRY_IMMEDIATELY; + } + + boolean isEos = (mBufferInfo.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0; + boolean hasSize = mBufferInfo.size > 0; + if (isEos) mIsDecoderEOS = true; + if (isEos || hasSize) { + mLastPresentationTimeUs = mBufferInfo.presentationTimeUs; + onDrainDecoder(mDecoder, + result, + mDecoderBuffers.getOutputBuffer(result), + mBufferInfo.presentationTimeUs, + isEos); + } + return DRAIN_STATE_CONSUMED; + } + + @SuppressWarnings("SameParameterValue") + private int drainEncoder(long timeoutUs) { + if (mIsEncoderEOS) return DRAIN_STATE_NONE; + + int result = mEncoder.dequeueOutputBuffer(mBufferInfo, timeoutUs); + switch (result) { + case MediaCodec.INFO_TRY_AGAIN_LATER: + return DRAIN_STATE_NONE; + case MediaCodec.INFO_OUTPUT_FORMAT_CHANGED: + onEncoderOutputFormatChanged(mEncoder, mEncoder.getOutputFormat()); + return DRAIN_STATE_SHOULD_RETRY_IMMEDIATELY; + case MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED: + mEncoderBuffers.onOutputBuffersChanged(); + return DRAIN_STATE_SHOULD_RETRY_IMMEDIATELY; + } + + if (mActualOutputFormat == null) { + throw new RuntimeException("Could not determine actual output format."); + } + + if ((mBufferInfo.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) { + mIsEncoderEOS = true; + mBufferInfo.set(0, 0, 0, mBufferInfo.flags); + } + if ((mBufferInfo.flags & MediaCodec.BUFFER_FLAG_CODEC_CONFIG) != 0) { + // SPS or PPS, which should be passed by MediaFormat. + mEncoder.releaseOutputBuffer(result, false); + return DRAIN_STATE_SHOULD_RETRY_IMMEDIATELY; + } + mMuxer.write(mTrackType, mEncoderBuffers.getOutputBuffer(result), mBufferInfo); + mEncoder.releaseOutputBuffer(result, false); + return DRAIN_STATE_CONSUMED; + } + + /** + * Called to drain the decoder. Implementors are required to call {@link MediaCodec#releaseOutputBuffer(int, boolean)} + * with the given bufferIndex at some point. + * + * @param decoder the decoder + * @param bufferIndex the buffer index to be released + * @param bufferData the buffer data + * @param presentationTimeUs frame timestamp + * @param endOfStream whether we are in end of stream + */ + protected abstract void onDrainDecoder(@NonNull MediaCodec decoder, + int bufferIndex, + @NonNull ByteBuffer bufferData, + long presentationTimeUs, + boolean endOfStream); + + + /** + * Called to feed the encoder with processed data. + * @param encoder the encoder + * @param timeoutUs a timeout for this op + * @return true if we want to keep working + */ + protected abstract boolean onFeedEncoder(@NonNull MediaCodec encoder, + @NonNull MediaCodecBuffers encoderBuffers, + long timeoutUs); +} diff --git a/lib/src/main/java/com/otaliastudios/transcoder/transcode/NoOpTrackTranscoder.java b/lib/src/main/java/com/otaliastudios/transcoder/transcode/NoOpTrackTranscoder.java index e1419955..60eecd3b 100644 --- a/lib/src/main/java/com/otaliastudios/transcoder/transcode/NoOpTrackTranscoder.java +++ b/lib/src/main/java/com/otaliastudios/transcoder/transcode/NoOpTrackTranscoder.java @@ -29,12 +29,12 @@ public void setUp(@NonNull MediaFormat desiredOutputFormat) { } @Override - public boolean stepPipeline() { + public boolean transcode() { return false; } @Override - public long getLastWrittenPresentationTime() { + public long getLastPresentationTime() { return 0; } diff --git a/lib/src/main/java/com/otaliastudios/transcoder/transcode/PassThroughTrackTranscoder.java b/lib/src/main/java/com/otaliastudios/transcoder/transcode/PassThroughTrackTranscoder.java index 029c23bd..1c88f8d3 100644 --- a/lib/src/main/java/com/otaliastudios/transcoder/transcode/PassThroughTrackTranscoder.java +++ b/lib/src/main/java/com/otaliastudios/transcoder/transcode/PassThroughTrackTranscoder.java @@ -24,6 +24,7 @@ import com.otaliastudios.transcoder.engine.TrackType; import com.otaliastudios.transcoder.engine.TranscoderMuxer; +import com.otaliastudios.transcoder.time.TimeInterpolator; import java.nio.ByteBuffer; import java.nio.ByteOrder; @@ -37,35 +38,41 @@ public class PassThroughTrackTranscoder implements TrackTranscoder { private int mBufferSize; private ByteBuffer mBuffer; private boolean mIsEOS; - private long mWrittenPresentationTimeUs; - private final MediaFormat mActualOutputFormat; - private boolean mFirstStepPipeline = true; + + private long mLastPresentationTime; + + private final MediaFormat mOutputFormat; + private boolean mOutputFormatSet = false; + + private TimeInterpolator mTimeInterpolator; public PassThroughTrackTranscoder(@NonNull MediaExtractor extractor, int trackIndex, @NonNull TranscoderMuxer muxer, - @NonNull TrackType trackType) { + @NonNull TrackType trackType, + @NonNull TimeInterpolator timeInterpolator) { mExtractor = extractor; mTrackIndex = trackIndex; mMuxer = muxer; mTrackType = trackType; - mActualOutputFormat = mExtractor.getTrackFormat(mTrackIndex); - mBufferSize = mActualOutputFormat.getInteger(MediaFormat.KEY_MAX_INPUT_SIZE); + mOutputFormat = mExtractor.getTrackFormat(mTrackIndex); + mBufferSize = mOutputFormat.getInteger(MediaFormat.KEY_MAX_INPUT_SIZE); mBuffer = ByteBuffer.allocateDirect(mBufferSize).order(ByteOrder.nativeOrder()); + + mTimeInterpolator = timeInterpolator; } @Override - public void setUp(@NonNull MediaFormat desiredOutputFormat) { - } + public void setUp(@NonNull MediaFormat desiredOutputFormat) { } @SuppressLint("Assert") @Override - public boolean stepPipeline() { + public boolean transcode() { if (mIsEOS) return false; - if (mFirstStepPipeline) { - mMuxer.setOutputFormat(mTrackType, mActualOutputFormat); - mFirstStepPipeline = false; + if (!mOutputFormatSet) { + mMuxer.setOutputFormat(mTrackType, mOutputFormat); + mOutputFormatSet = true; } int trackIndex = mExtractor.getSampleTrackIndex(); if (trackIndex < 0) { @@ -82,17 +89,19 @@ public boolean stepPipeline() { assert sampleSize <= mBufferSize; boolean isKeyFrame = (mExtractor.getSampleFlags() & MediaExtractor.SAMPLE_FLAG_SYNC) != 0; int flags = isKeyFrame ? MediaCodec.BUFFER_FLAG_SYNC_FRAME : 0; - mBufferInfo.set(0, sampleSize, mExtractor.getSampleTime(), flags); + long realTimestampUs = mExtractor.getSampleTime(); + long timestampUs = mTimeInterpolator.interpolate(mTrackType, realTimestampUs); + mBufferInfo.set(0, sampleSize, timestampUs, flags); mMuxer.write(mTrackType, mBuffer, mBufferInfo); - mWrittenPresentationTimeUs = mBufferInfo.presentationTimeUs; + mLastPresentationTime = realTimestampUs; mExtractor.advance(); return true; } @Override - public long getLastWrittenPresentationTime() { - return mWrittenPresentationTimeUs; + public long getLastPresentationTime() { + return mLastPresentationTime; } @Override diff --git a/lib/src/main/java/com/otaliastudios/transcoder/transcode/TrackTranscoder.java b/lib/src/main/java/com/otaliastudios/transcoder/transcode/TrackTranscoder.java index 9e90e48a..7be88bd4 100644 --- a/lib/src/main/java/com/otaliastudios/transcoder/transcode/TrackTranscoder.java +++ b/lib/src/main/java/com/otaliastudios/transcoder/transcode/TrackTranscoder.java @@ -25,19 +25,20 @@ public interface TrackTranscoder { void setUp(@NonNull MediaFormat desiredOutputFormat); /** - * Step pipeline if output is available in any step of it. + * Perform transcoding if output is available in any step of it. * It assumes muxer has been started, so you should call muxer.start() first. * * @return true if data moved in pipeline. */ - boolean stepPipeline(); + boolean transcode(); /** - * Get presentation time of last sample written to muxer. + * Get presentation time of last sample taken from encoder. + * This presentation time should not be affected by {@link com.otaliastudios.transcoder.time.TimeInterpolator}s. * - * @return Presentation time in micro-second. Return value is undefined if finished writing. + * @return Presentation time in microseconds. Return value is undefined if finished writing. */ - long getLastWrittenPresentationTime(); + long getLastPresentationTime(); boolean isFinished(); diff --git a/lib/src/main/java/com/otaliastudios/transcoder/transcode/VideoTrackTranscoder.java b/lib/src/main/java/com/otaliastudios/transcoder/transcode/VideoTrackTranscoder.java index a472c721..c58440d7 100644 --- a/lib/src/main/java/com/otaliastudios/transcoder/transcode/VideoTrackTranscoder.java +++ b/lib/src/main/java/com/otaliastudios/transcoder/transcode/VideoTrackTranscoder.java @@ -24,93 +24,68 @@ import com.otaliastudios.transcoder.engine.TrackType; import com.otaliastudios.transcoder.engine.TranscoderMuxer; import com.otaliastudios.transcoder.internal.MediaCodecBuffers; +import com.otaliastudios.transcoder.time.TimeInterpolator; import com.otaliastudios.transcoder.transcode.internal.VideoDecoderOutput; import com.otaliastudios.transcoder.transcode.internal.VideoEncoderInput; import com.otaliastudios.transcoder.internal.Logger; import com.otaliastudios.transcoder.internal.MediaFormatConstants; +import com.otaliastudios.transcoder.transcode.internal.VideoFrameDropper; -import java.io.IOException; +import java.nio.ByteBuffer; // Refer: https://android.googlesource.com/platform/cts/+/lollipop-release/tests/tests/media/src/android/media/cts/ExtractDecodeEditEncodeMuxTest.java -public class VideoTrackTranscoder implements TrackTranscoder { +public class VideoTrackTranscoder extends BaseTrackTranscoder { private static final String TAG = VideoTrackTranscoder.class.getSimpleName(); + @SuppressWarnings("unused") private static final Logger LOG = new Logger(TAG); - private static final int DRAIN_STATE_NONE = 0; - private static final int DRAIN_STATE_SHOULD_RETRY_IMMEDIATELY = 1; - private static final int DRAIN_STATE_CONSUMED = 2; - - private final MediaExtractor mExtractor; - private final int mTrackIndex; - private final TranscoderMuxer mMuxer; - private final MediaCodec.BufferInfo mBufferInfo = new MediaCodec.BufferInfo(); - private MediaCodec mDecoder; - private MediaCodec mEncoder; - private MediaCodecBuffers mDecoderBuffers; - private MediaCodecBuffers mEncoderBuffers; - - private MediaFormat mActualOutputFormat; - private boolean mIsExtractorEOS; - private boolean mIsDecoderEOS; - private boolean mIsEncoderEOS; - private boolean mDecoderStarted; - private boolean mEncoderStarted; - private long mWrittenPresentationTimeUs; - private VideoDecoderOutput mDecoderOutputSurface; private VideoEncoderInput mEncoderInputSurface; - - private double mInFrameRateReciprocal; - private double mOutFrameRateReciprocal; - private double mFrameRateReciprocalSum; + private MediaCodec mEncoder; // Keep this since we want to signal EOS on it. + private VideoFrameDropper mFrameDropper; + private TimeInterpolator mTimeInterpolator; public VideoTrackTranscoder( @NonNull MediaExtractor extractor, + @NonNull TranscoderMuxer muxer, int trackIndex, - @NonNull TranscoderMuxer muxer) { - mExtractor = extractor; - mTrackIndex = trackIndex; - mMuxer = muxer; + @NonNull TimeInterpolator timeInterpolator) { + super(extractor, muxer, TrackType.VIDEO, trackIndex); + mTimeInterpolator = timeInterpolator; } @Override - public void setUp(@NonNull MediaFormat desiredOutputFormat) { - mExtractor.selectTrack(mTrackIndex); - - MediaFormat trackFormat = mExtractor.getTrackFormat(mTrackIndex); - int inFrameRate = trackFormat.getInteger(MediaFormat.KEY_FRAME_RATE); - int outFrameRate = desiredOutputFormat.getInteger(MediaFormat.KEY_FRAME_RATE); - mInFrameRateReciprocal = 1.0d / inFrameRate; - mOutFrameRateReciprocal = 1.0d / outFrameRate; - LOG.v("mInFrameRateReciprocal: " + mInFrameRateReciprocal + " mOutFrameRateReciprocal: " + mOutFrameRateReciprocal); - - // Configure encoder. - try { - mEncoder = MediaCodec.createEncoderByType(desiredOutputFormat.getString(MediaFormat.KEY_MIME)); - } catch (IOException e) { - throw new IllegalStateException(e); - } - mEncoder.configure(desiredOutputFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE); - mEncoderInputSurface = new VideoEncoderInput(mEncoder.createInputSurface()); - mEncoder.start(); - mEncoderStarted = true; - mEncoderBuffers = new MediaCodecBuffers(mEncoder); + protected void onStartEncoder(@NonNull MediaFormat format, @NonNull MediaCodec encoder) { + mEncoderInputSurface = new VideoEncoderInput(encoder.createInputSurface()); + super.onStartEncoder(format, encoder); + } - // Configure decoder. - MediaFormat inputFormat = mExtractor.getTrackFormat(mTrackIndex); - if (inputFormat.containsKey(MediaFormatConstants.KEY_ROTATION_DEGREES)) { - // Decoded video is rotated automatically in Android 5.0 lollipop. - // Turn off here because we don't want to encode rotated one. + @Override + protected void onConfigureDecoder(@NonNull MediaFormat format, @NonNull MediaCodec decoder) { + if (format.containsKey(MediaFormatConstants.KEY_ROTATION_DEGREES)) { + // Decoded video is rotated automatically in Android 5.0 lollipop. Turn off here because we don't want to encode rotated one. // refer: https://android.googlesource.com/platform/frameworks/av/+blame/lollipop-release/media/libstagefright/Utils.cpp - inputFormat.setInteger(MediaFormatConstants.KEY_ROTATION_DEGREES, 0); + format.setInteger(MediaFormatConstants.KEY_ROTATION_DEGREES, 0); } + mDecoderOutputSurface = new VideoDecoderOutput(); + decoder.configure(format, mDecoderOutputSurface.getSurface(), null, 0); + } + + @Override + protected void onCodecsStarted(@NonNull MediaFormat inputFormat, @NonNull MediaFormat outputFormat, @NonNull MediaCodec decoder, @NonNull MediaCodec encoder) { + super.onCodecsStarted(inputFormat, outputFormat, decoder, encoder); + mFrameDropper = VideoFrameDropper.newDropper( + inputFormat.getInteger(MediaFormat.KEY_FRAME_RATE), + outputFormat.getInteger(MediaFormat.KEY_FRAME_RATE)); + mEncoder = encoder; + // Cropping support. float inputWidth = inputFormat.getInteger(MediaFormat.KEY_WIDTH); float inputHeight = inputFormat.getInteger(MediaFormat.KEY_HEIGHT); float inputRatio = inputWidth / inputHeight; - float outputWidth = desiredOutputFormat.getInteger(MediaFormat.KEY_WIDTH); - float outputHeight = desiredOutputFormat.getInteger(MediaFormat.KEY_HEIGHT); + float outputWidth = outputFormat.getInteger(MediaFormat.KEY_WIDTH); + float outputHeight = outputFormat.getInteger(MediaFormat.KEY_HEIGHT); float outputRatio = outputWidth / outputHeight; float scaleX = 1, scaleY = 1; if (inputRatio > outputRatio) { // Input wider. We have a scaleX. @@ -120,44 +95,9 @@ public void setUp(@NonNull MediaFormat desiredOutputFormat) { } // I don't think we should consider rotation and flip these - we operate on non-rotated // surfaces and pass the input rotation metadata to the output muxer, see TranscoderEngine.setupMetadata. - mDecoderOutputSurface = new VideoDecoderOutput(scaleX, scaleY); - try { - mDecoder = MediaCodec.createDecoderByType(inputFormat.getString(MediaFormat.KEY_MIME)); - } catch (IOException e) { - throw new IllegalStateException(e); - } - mDecoder.configure(inputFormat, mDecoderOutputSurface.getSurface(), null, 0); - mDecoder.start(); - mDecoderStarted = true; - mDecoderBuffers = new MediaCodecBuffers(mDecoder); - } - - @Override - public boolean stepPipeline() { - boolean busy = false; - int status; - while (drainEncoder(0) != DRAIN_STATE_NONE) busy = true; - do { - status = drainDecoder(0); - if (status != DRAIN_STATE_NONE) busy = true; - // NOTE: not repeating to keep from deadlock when encoder is full. - } while (status == DRAIN_STATE_SHOULD_RETRY_IMMEDIATELY); - while (drainExtractor(0) != DRAIN_STATE_NONE) busy = true; - - return busy; + mDecoderOutputSurface.setScale(scaleX, scaleY); } - @Override - public long getLastWrittenPresentationTime() { - return mWrittenPresentationTimeUs; - } - - @Override - public boolean isFinished() { - return mIsEncoderEOS; - } - - // TODO: CloseGuard @Override public void release() { if (mDecoderOutputSurface != null) { @@ -168,120 +108,30 @@ public void release() { mEncoderInputSurface.release(); mEncoderInputSurface = null; } - if (mDecoder != null) { - if (mDecoderStarted) mDecoder.stop(); - mDecoder.release(); - mDecoder = null; - } - if (mEncoder != null) { - if (mEncoderStarted) mEncoder.stop(); - mEncoder.release(); - mEncoder = null; - } - } - - @SuppressWarnings("SameParameterValue") - private int drainExtractor(long timeoutUs) { - if (mIsExtractorEOS) return DRAIN_STATE_NONE; - int trackIndex = mExtractor.getSampleTrackIndex(); - if (trackIndex >= 0 && trackIndex != mTrackIndex) { - return DRAIN_STATE_NONE; - } - int result = mDecoder.dequeueInputBuffer(timeoutUs); - if (result < 0) return DRAIN_STATE_NONE; - if (trackIndex < 0) { - mIsExtractorEOS = true; - mDecoder.queueInputBuffer(result, 0, 0, 0, MediaCodec.BUFFER_FLAG_END_OF_STREAM); - return DRAIN_STATE_NONE; - } - int sampleSize = mExtractor.readSampleData(mDecoderBuffers.getInputBuffer(result), 0); - boolean isKeyFrame = (mExtractor.getSampleFlags() & MediaExtractor.SAMPLE_FLAG_SYNC) != 0; - mDecoder.queueInputBuffer(result, 0, sampleSize, mExtractor.getSampleTime(), isKeyFrame ? MediaCodec.BUFFER_FLAG_SYNC_FRAME : 0); - mExtractor.advance(); - return DRAIN_STATE_CONSUMED; - } - - @SuppressWarnings("SameParameterValue") - private int drainDecoder(long timeoutUs) { - if (mIsDecoderEOS) return DRAIN_STATE_NONE; - int result = mDecoder.dequeueOutputBuffer(mBufferInfo, timeoutUs); - switch (result) { - case MediaCodec.INFO_TRY_AGAIN_LATER: - return DRAIN_STATE_NONE; - case MediaCodec.INFO_OUTPUT_FORMAT_CHANGED: - case MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED: - return DRAIN_STATE_SHOULD_RETRY_IMMEDIATELY; - } - if ((mBufferInfo.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) { - mEncoder.signalEndOfInputStream(); - mIsDecoderEOS = true; - mBufferInfo.size = 0; - } - boolean doRender = shouldRenderFrame(); - // NOTE: doRender will block if buffer (of encoder) is full. - // Refer: http://bigflake.com/mediacodec/CameraToMpegTest.java.txt - mDecoder.releaseOutputBuffer(result, doRender); - if (doRender) { - mDecoderOutputSurface.drawFrame(); - mEncoderInputSurface.onFrame(mBufferInfo.presentationTimeUs); - } - return DRAIN_STATE_CONSUMED; + super.release(); + mEncoder = null; } - //Refer:https://stackoverflow.com/questions/4223766/dropping-video-frames - private boolean shouldRenderFrame() { - if (mBufferInfo.size <= 0) return false; - boolean firstFrame = Double.valueOf(0d).equals(mFrameRateReciprocalSum); - mFrameRateReciprocalSum += mInFrameRateReciprocal; - if (firstFrame) { - // render frame - LOG.v("render this frame -> mFrameRateReciprocalSum: " + mFrameRateReciprocalSum); - return true; - } - if (mFrameRateReciprocalSum > mOutFrameRateReciprocal) { - mFrameRateReciprocalSum -= mOutFrameRateReciprocal; - // render frame - LOG.v("render this frame -> mFrameRateReciprocalSum: " + mFrameRateReciprocalSum); - return true; - } - // drop frame - LOG.v("drop this frame -> mFrameRateReciprocalSum: " + mFrameRateReciprocalSum); + @Override + protected boolean onFeedEncoder(@NonNull MediaCodec encoder, @NonNull MediaCodecBuffers encoderBuffers, long timeoutUs) { + // We do not feed the encoder, instead we wait for the encoder surface onFrameAvailable callback. return false; } - @SuppressWarnings("SameParameterValue") - private int drainEncoder(long timeoutUs) { - if (mIsEncoderEOS) return DRAIN_STATE_NONE; - int result = mEncoder.dequeueOutputBuffer(mBufferInfo, timeoutUs); - switch (result) { - case MediaCodec.INFO_TRY_AGAIN_LATER: - return DRAIN_STATE_NONE; - case MediaCodec.INFO_OUTPUT_FORMAT_CHANGED: - if (mActualOutputFormat != null) - throw new RuntimeException("Video output format changed twice."); - mActualOutputFormat = mEncoder.getOutputFormat(); - mMuxer.setOutputFormat(TrackType.VIDEO, mActualOutputFormat); - return DRAIN_STATE_SHOULD_RETRY_IMMEDIATELY; - case MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED: - mEncoderBuffers.onOutputBuffersChanged(); - return DRAIN_STATE_SHOULD_RETRY_IMMEDIATELY; - } - if (mActualOutputFormat == null) { - throw new RuntimeException("Could not determine actual output format."); - } - - if ((mBufferInfo.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) { - mIsEncoderEOS = true; - mBufferInfo.set(0, 0, 0, mBufferInfo.flags); - } - if ((mBufferInfo.flags & MediaCodec.BUFFER_FLAG_CODEC_CONFIG) != 0) { - // SPS or PPS, which should be passed by MediaFormat. - mEncoder.releaseOutputBuffer(result, false); - return DRAIN_STATE_SHOULD_RETRY_IMMEDIATELY; + @Override + protected void onDrainDecoder(@NonNull MediaCodec decoder, int bufferIndex, @NonNull ByteBuffer bufferData, long presentationTimeUs, boolean endOfStream) { + if (endOfStream) { + mEncoder.signalEndOfInputStream(); + decoder.releaseOutputBuffer(bufferIndex, false); + } else { + long interpolatedTimeUs = mTimeInterpolator.interpolate(TrackType.VIDEO, presentationTimeUs); + if (mFrameDropper.shouldRenderFrame(interpolatedTimeUs)) { + decoder.releaseOutputBuffer(bufferIndex, true); + mDecoderOutputSurface.drawFrame(); + mEncoderInputSurface.onFrame(interpolatedTimeUs); + } else { + decoder.releaseOutputBuffer(bufferIndex, false); + } } - mMuxer.write(TrackType.VIDEO, mEncoderBuffers.getOutputBuffer(result), mBufferInfo); - mWrittenPresentationTimeUs = mBufferInfo.presentationTimeUs; - mEncoder.releaseOutputBuffer(result, false); - return DRAIN_STATE_CONSUMED; } } diff --git a/lib/src/main/java/com/otaliastudios/transcoder/transcode/internal/AudioBuffer.java b/lib/src/main/java/com/otaliastudios/transcoder/transcode/internal/AudioBuffer.java new file mode 100644 index 00000000..0f8150b8 --- /dev/null +++ b/lib/src/main/java/com/otaliastudios/transcoder/transcode/internal/AudioBuffer.java @@ -0,0 +1,10 @@ +package com.otaliastudios.transcoder.transcode.internal; + +import java.nio.ShortBuffer; + +class AudioBuffer { + int decoderBufferIndex = -1; + long decoderTimestampUs = 0; + ShortBuffer decoderData = null; + boolean isEndOfStream = false; +} diff --git a/lib/src/main/java/com/otaliastudios/transcoder/transcode/internal/AudioChannel.java b/lib/src/main/java/com/otaliastudios/transcoder/transcode/internal/AudioChannel.java deleted file mode 100644 index 19ce52e9..00000000 --- a/lib/src/main/java/com/otaliastudios/transcoder/transcode/internal/AudioChannel.java +++ /dev/null @@ -1,241 +0,0 @@ -package com.otaliastudios.transcoder.transcode.internal; - -import android.media.MediaCodec; -import android.media.MediaFormat; - -import androidx.annotation.NonNull; - -import com.otaliastudios.transcoder.internal.MediaCodecBuffers; -import com.otaliastudios.transcoder.remix.AudioRemixer; - -import java.nio.ByteBuffer; -import java.nio.ByteOrder; -import java.nio.ShortBuffer; -import java.util.ArrayDeque; -import java.util.Queue; - -/** - * Channel of raw audio from decoder to encoder. - * Performs the necessary conversion between different input & output audio formats. - * - * We currently support upmixing from mono to stereo & downmixing from stereo to mono. - * Sample rate conversion is not supported yet. - */ -public class AudioChannel { - - private static class AudioBuffer { - int bufferIndex; - long presentationTimeUs; - ShortBuffer data; - } - - public static final int BUFFER_INDEX_END_OF_STREAM = -1; - - private static final int BYTES_PER_SHORT = 2; - private static final long MICROSECS_PER_SEC = 1000000; - - private final Queue mEmptyBuffers = new ArrayDeque<>(); - private final Queue mFilledBuffers = new ArrayDeque<>(); - - private final MediaCodec mDecoder; - private final MediaCodec mEncoder; - private final MediaFormat mEncodeFormat; - - private int mInputSampleRate; - private int mInputChannelCount; - private int mOutputChannelCount; - - private AudioRemixer mRemixer; - - private final MediaCodecBuffers mDecoderBuffers; - private final MediaCodecBuffers mEncoderBuffers; - - private final AudioBuffer mOverflowBuffer = new AudioBuffer(); - - private MediaFormat mActualDecodedFormat; - - public AudioChannel(@NonNull final MediaCodec decoder, - @NonNull final MediaCodec encoder, - @NonNull final MediaFormat encodeFormat) { - mDecoder = decoder; - mEncoder = encoder; - mEncodeFormat = encodeFormat; - - mDecoderBuffers = new MediaCodecBuffers(mDecoder); - mEncoderBuffers = new MediaCodecBuffers(mEncoder); - } - - public void setActualDecodedFormat(@NonNull final MediaFormat decodedFormat) { - mActualDecodedFormat = decodedFormat; - - // TODO I think these exceptions are either useless or not in the right place. - // We have MediaFormatValidator doing this kind of stuff. - - mInputSampleRate = mActualDecodedFormat.getInteger(MediaFormat.KEY_SAMPLE_RATE); - if (mInputSampleRate != mEncodeFormat.getInteger(MediaFormat.KEY_SAMPLE_RATE)) { - throw new UnsupportedOperationException("Audio sample rate conversion not supported yet."); - } - - mInputChannelCount = mActualDecodedFormat.getInteger(MediaFormat.KEY_CHANNEL_COUNT); - mOutputChannelCount = mEncodeFormat.getInteger(MediaFormat.KEY_CHANNEL_COUNT); - - if (mInputChannelCount != 1 && mInputChannelCount != 2) { - throw new UnsupportedOperationException("Input channel count (" + mInputChannelCount + ") not supported."); - } - - if (mOutputChannelCount != 1 && mOutputChannelCount != 2) { - throw new UnsupportedOperationException("Output channel count (" + mOutputChannelCount + ") not supported."); - } - - if (mInputChannelCount > mOutputChannelCount) { - mRemixer = AudioRemixer.DOWNMIX; - } else if (mInputChannelCount < mOutputChannelCount) { - mRemixer = AudioRemixer.UPMIX; - } else { - mRemixer = AudioRemixer.PASSTHROUGH; - } - - mOverflowBuffer.presentationTimeUs = 0; - } - - public void drainDecoderBufferAndQueue(final int bufferIndex, final long presentationTimeUs) { - if (mActualDecodedFormat == null) { - throw new RuntimeException("Buffer received before format!"); - } - - final ByteBuffer data = bufferIndex == BUFFER_INDEX_END_OF_STREAM ? - null : mDecoderBuffers.getOutputBuffer(bufferIndex); - - AudioBuffer buffer = mEmptyBuffers.poll(); - if (buffer == null) { - buffer = new AudioBuffer(); - } - - buffer.bufferIndex = bufferIndex; - buffer.presentationTimeUs = presentationTimeUs; - buffer.data = data == null ? null : data.asShortBuffer(); - - if (mOverflowBuffer.data == null) { - // data should be null only on BUFFER_INDEX_END_OF_STREAM - //noinspection ConstantConditions - mOverflowBuffer.data = ByteBuffer - .allocateDirect(data.capacity()) - .order(ByteOrder.nativeOrder()) - .asShortBuffer(); - mOverflowBuffer.data.clear().flip(); - } - - mFilledBuffers.add(buffer); - } - - public boolean feedEncoder(@SuppressWarnings("SameParameterValue") long timeoutUs) { - final boolean hasOverflow = mOverflowBuffer.data != null && mOverflowBuffer.data.hasRemaining(); - if (mFilledBuffers.isEmpty() && !hasOverflow) { - // No audio data - Bail out - return false; - } - - final int encoderInBuffIndex = mEncoder.dequeueInputBuffer(timeoutUs); - if (encoderInBuffIndex < 0) { - // Encoder is full - Bail out - return false; - } - - // Drain overflow first - final ShortBuffer outBuffer = mEncoderBuffers.getInputBuffer(encoderInBuffIndex).asShortBuffer(); - if (hasOverflow) { - final long presentationTimeUs = drainOverflow(outBuffer); - mEncoder.queueInputBuffer(encoderInBuffIndex, - 0, outBuffer.position() * BYTES_PER_SHORT, - presentationTimeUs, 0); - return true; - } - - final AudioBuffer inBuffer = mFilledBuffers.poll(); - // At this point inBuffer is not null, because we checked mFilledBuffers.isEmpty() - // and we don't have overflow (if we had, we got out). - //noinspection ConstantConditions - if (inBuffer.bufferIndex == BUFFER_INDEX_END_OF_STREAM) { - mEncoder.queueInputBuffer(encoderInBuffIndex, 0, 0, 0, MediaCodec.BUFFER_FLAG_END_OF_STREAM); - return false; - } - - final long presentationTimeUs = remixAndMaybeFillOverflow(inBuffer, outBuffer); - mEncoder.queueInputBuffer(encoderInBuffIndex, - 0, outBuffer.position() * BYTES_PER_SHORT, - presentationTimeUs, 0); - mDecoder.releaseOutputBuffer(inBuffer.bufferIndex, false); - mEmptyBuffers.add(inBuffer); - return true; - } - - private static long sampleCountToDurationUs( - final int sampleCount, - final int sampleRate, - final int channelCount) { - return (sampleCount / (sampleRate * MICROSECS_PER_SEC)) / channelCount; - } - - private long drainOverflow(@NonNull final ShortBuffer outBuff) { - final ShortBuffer overflowBuff = mOverflowBuffer.data; - final int overflowLimit = overflowBuff.limit(); - final int overflowSize = overflowBuff.remaining(); - - final long beginPresentationTimeUs = mOverflowBuffer.presentationTimeUs + - sampleCountToDurationUs(overflowBuff.position(), - mInputSampleRate, - mOutputChannelCount); - - outBuff.clear(); - // Limit overflowBuff to outBuff's capacity - overflowBuff.limit(outBuff.capacity()); - // Load overflowBuff onto outBuff - outBuff.put(overflowBuff); - - if (overflowSize >= outBuff.capacity()) { - // Overflow fully consumed - Reset - overflowBuff.clear().limit(0); - } else { - // Only partially consumed - Keep position & restore previous limit - overflowBuff.limit(overflowLimit); - } - - return beginPresentationTimeUs; - } - - private long remixAndMaybeFillOverflow(final AudioBuffer input, - final ShortBuffer outBuff) { - final ShortBuffer inBuff = input.data; - final ShortBuffer overflowBuff = mOverflowBuffer.data; - - outBuff.clear(); - - // Reset position to 0, and set limit to capacity (Since MediaCodec doesn't do that for us) - inBuff.clear(); - - if (inBuff.remaining() > outBuff.remaining()) { - // Overflow - // Limit inBuff to outBuff's capacity - inBuff.limit(outBuff.capacity()); - mRemixer.remix(inBuff, outBuff); - - // Reset limit to its own capacity & Keep position - inBuff.limit(inBuff.capacity()); - - // Remix the rest onto overflowBuffer - // NOTE: We should only reach this point when overflow buffer is empty - final long consumedDurationUs = - sampleCountToDurationUs(inBuff.position(), mInputSampleRate, mInputChannelCount); - mRemixer.remix(inBuff, overflowBuff); - - // Seal off overflowBuff & mark limit - overflowBuff.flip(); - mOverflowBuffer.presentationTimeUs = input.presentationTimeUs + consumedDurationUs; - } else { - // No overflow - mRemixer.remix(inBuff, outBuff); - } - - return input.presentationTimeUs; - } -} diff --git a/lib/src/main/java/com/otaliastudios/transcoder/transcode/internal/AudioConversions.java b/lib/src/main/java/com/otaliastudios/transcoder/transcode/internal/AudioConversions.java new file mode 100644 index 00000000..be0df1b8 --- /dev/null +++ b/lib/src/main/java/com/otaliastudios/transcoder/transcode/internal/AudioConversions.java @@ -0,0 +1,34 @@ +package com.otaliastudios.transcoder.transcode.internal; + +class AudioConversions { + + private static final int BYTES_PER_SAMPLE_PER_CHANNEL = 2; // Assuming 16bit audio, so 2 + private static final long MICROSECONDS_PER_SECOND = 1000000L; + private static final int BYTES_PER_SHORT = 2; + + @SuppressWarnings("WeakerAccess") + static long bytesToUs( + int bytes /* bytes */, + int sampleRate /* samples/sec */, + int channels /* channel */ + ) { + int byteRatePerChannel = sampleRate * BYTES_PER_SAMPLE_PER_CHANNEL; // bytes/sec/channel + int byteRate = byteRatePerChannel * channels; // bytes/sec + return MICROSECONDS_PER_SECOND * bytes / byteRate; // usec + } + + @SuppressWarnings("WeakerAccess") + static int usToBytes(long us, int sampleRate, int channels) { + int byteRatePerChannel = sampleRate * BYTES_PER_SAMPLE_PER_CHANNEL; + int byteRate = byteRatePerChannel * channels; + return (int) Math.ceil((double) us * byteRate / MICROSECONDS_PER_SECOND); + } + + static long shortsToUs(int shorts, int sampleRate, int channels) { + return bytesToUs(shorts * BYTES_PER_SHORT, sampleRate, channels); + } + + static int usToShorts(long us, int sampleRate, int channels) { + return usToBytes(us, sampleRate, channels) / BYTES_PER_SHORT; + } +} diff --git a/lib/src/main/java/com/otaliastudios/transcoder/transcode/internal/AudioEngine.java b/lib/src/main/java/com/otaliastudios/transcoder/transcode/internal/AudioEngine.java new file mode 100644 index 00000000..da57cf23 --- /dev/null +++ b/lib/src/main/java/com/otaliastudios/transcoder/transcode/internal/AudioEngine.java @@ -0,0 +1,289 @@ +package com.otaliastudios.transcoder.transcode.internal; + +import android.media.MediaCodec; +import android.media.MediaFormat; + +import androidx.annotation.NonNull; + +import com.otaliastudios.transcoder.engine.TrackType; +import com.otaliastudios.transcoder.internal.Logger; +import com.otaliastudios.transcoder.internal.MediaCodecBuffers; +import com.otaliastudios.transcoder.remix.AudioRemixer; +import com.otaliastudios.transcoder.stretch.AudioStretcher; +import com.otaliastudios.transcoder.time.TimeInterpolator; + +import java.nio.Buffer; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.ShortBuffer; +import java.util.ArrayDeque; +import java.util.Queue; + +/** + * Channel of raw audio from decoder to encoder. + * Performs the necessary conversion between different input & output audio formats. + * + * We currently support upmixing from mono to stereo & downmixing from stereo to mono. + * Sample rate conversion is not supported yet. + */ +public class AudioEngine { + + private static final int BYTES_PER_SHORT = 2; + + private static final String TAG = AudioEngine.class.getSimpleName(); + private static final Logger LOG = new Logger(TAG); + + private final Queue mEmptyBuffers = new ArrayDeque<>(); + private final Queue mPendingBuffers = new ArrayDeque<>(); + private final MediaCodec mDecoder; + private final MediaCodec mEncoder; + private final int mSampleRate; + private final int mDecoderChannels; + @SuppressWarnings("FieldCanBeLocal") private final int mEncoderChannels; + private final AudioRemixer mRemixer; + private final AudioStretcher mStretcher; + private final TimeInterpolator mTimeInterpolator; + private long mLastDecoderUs = Long.MIN_VALUE; + private long mLastEncoderUs = Long.MIN_VALUE; + private ShortBuffer mTempBuffer; + + /** + * The AudioEngine should be created when we know the actual decoded format, + * which means that the decoder has reached {@link MediaCodec#INFO_OUTPUT_FORMAT_CHANGED}. + * + * @param decoder a decoder + * @param decoderOutputFormat the decoder output format + * @param encoder an encoder + * @param encoderOutputFormat the encoder output format + */ + public AudioEngine(@NonNull MediaCodec decoder, + @NonNull MediaFormat decoderOutputFormat, + @NonNull MediaCodec encoder, + @NonNull MediaFormat encoderOutputFormat, + @NonNull TimeInterpolator timeInterpolator, + @NonNull AudioStretcher audioStretcher) { + mDecoder = decoder; + mEncoder = encoder; + mTimeInterpolator = timeInterpolator; + + // Get and check sample rate. + int outputSampleRate = encoderOutputFormat.getInteger(MediaFormat.KEY_SAMPLE_RATE); + int inputSampleRate = decoderOutputFormat.getInteger(MediaFormat.KEY_SAMPLE_RATE); + if (inputSampleRate != outputSampleRate) { + throw new UnsupportedOperationException("Audio sample rate conversion not supported yet."); + } + mSampleRate = inputSampleRate; + + // Check channel count. + mEncoderChannels = encoderOutputFormat.getInteger(MediaFormat.KEY_CHANNEL_COUNT); + mDecoderChannels = decoderOutputFormat.getInteger(MediaFormat.KEY_CHANNEL_COUNT); + if (mEncoderChannels != 1 && mEncoderChannels != 2) { + throw new UnsupportedOperationException("Output channel count (" + mEncoderChannels + ") not supported."); + } + if (mDecoderChannels != 1 && mDecoderChannels != 2) { + throw new UnsupportedOperationException("Input channel count (" + mDecoderChannels + ") not supported."); + } + + // Create remixer and stretcher. + if (mDecoderChannels > mEncoderChannels) { + mRemixer = AudioRemixer.DOWNMIX; + } else if (mDecoderChannels < mEncoderChannels) { + mRemixer = AudioRemixer.UPMIX; + } else { + mRemixer = AudioRemixer.PASSTHROUGH; + } + mStretcher = audioStretcher; + } + + /** + * Returns true if we have raw buffers to be processed. + * @return true if we have + */ + private boolean hasPendingBuffers() { + return !mPendingBuffers.isEmpty(); + } + + /** + * Drains the decoder, which means scheduling data for processing. + * We fill a new {@link AudioBuffer} with data (not a copy, just a view of it) + * and add it to the filled buffers queue. + * + * @param bufferIndex the buffer index + * @param bufferData the buffer data + * @param presentationTimeUs the presentation time + * @param endOfStream true if end of stream + */ + public void drainDecoder(final int bufferIndex, + @NonNull ByteBuffer bufferData, + final long presentationTimeUs, + final boolean endOfStream) { + if (mRemixer == null) throw new RuntimeException("Buffer received before format!"); + AudioBuffer buffer = mEmptyBuffers.poll(); + if (buffer == null) buffer = new AudioBuffer(); + buffer.decoderBufferIndex = bufferIndex; + buffer.decoderTimestampUs = endOfStream ? 0 : presentationTimeUs; + buffer.decoderData = endOfStream ? null : bufferData.asShortBuffer(); + buffer.isEndOfStream = endOfStream; + mPendingBuffers.add(buffer); + } + + /** + * Feeds the encoder, which in our case means processing a filled buffer, + * then releasing it and adding it back to {@link #mEmptyBuffers}. + * + * @param encoderBuffers the encoder buffers + * @param timeoutUs a timeout for this operation + * @return true if we want to keep working + */ + public boolean feedEncoder(@NonNull MediaCodecBuffers encoderBuffers, long timeoutUs) { + if (!hasPendingBuffers()) return false; + + // First of all, see if encoder has buffers that we can write into. + // If we don't have an output buffer, there's nothing we can do. + final int encoderBufferIndex = mEncoder.dequeueInputBuffer(timeoutUs); + if (encoderBufferIndex < 0) return false; + ShortBuffer encoderBuffer = encoderBuffers.getInputBuffer(encoderBufferIndex).asShortBuffer(); + encoderBuffer.clear(); + + // Get the latest raw buffer to be processed. + AudioBuffer buffer = mPendingBuffers.peek(); + + // When endOfStream, just signal EOS and return false. + //noinspection ConstantConditions + if (buffer.isEndOfStream) { + mEncoder.queueInputBuffer(encoderBufferIndex, + 0, + 0, + 0, + MediaCodec.BUFFER_FLAG_END_OF_STREAM); + return false; + } + + // Process the buffer. + boolean overflows = process(buffer, encoderBuffer, encoderBufferIndex); + if (overflows) { + // If this buffer does overflow, we will keep it in the queue and do + // not release. It will be used at the next cycle (presumably soon, + // since we return true). + return true; + } else { + // If this buffer does not overflow, it can be removed from our queue, + // re-added to empty, and also released from the decoder buffers. + mPendingBuffers.remove(); + mEmptyBuffers.add(buffer); + mDecoder.releaseOutputBuffer(buffer.decoderBufferIndex, false); + return true; + } + } + + /** + * Processes a pending buffer. + * + * We have an output buffer of restricted size, and our input size can be already bigger + * or grow after a {@link TimeInterpolator} operation or a {@link AudioRemixer} one. + * + * For this reason, we instead start from the output size and compute the input size that + * would generate an output of that size. We then select this part of the input and only + * process. + * + * If input is restricted, this means that the input buffer must be processed again. + * We will return true in this case. At the next cycle, the input buffer should be identical + * to the previous, but: + * + * - {@link Buffer#position()} will be increased to exclude the already processed values + * - {@link AudioBuffer#decoderTimestampUs} will be increased to include the already processed values + * + * So everything should work as expected for repeated cycles. + * + * Before returning, this function should release the encoder buffer using + * {@link MediaCodec#queueInputBuffer(int, int, int, long, int)}. + * + * @param buffer coming from decoder. Has valid limit and position + * @param encoderBuffer coming from encoder. At this point this is in a cleared state + * @param encoderBufferIndex the index of encoderBuffer so we can release it + */ + private boolean process(@NonNull AudioBuffer buffer, @NonNull ShortBuffer encoderBuffer, int encoderBufferIndex) { + // Only process the amount of data that can fill in the encoderBuffer. + final int outputSize = encoderBuffer.remaining(); + final int inputSize = buffer.decoderData.remaining(); + int processedInputSize = inputSize; + + // 1. Perform TimeInterpolator computation + long encoderUs = mTimeInterpolator.interpolate(TrackType.AUDIO, buffer.decoderTimestampUs); + if (mLastDecoderUs == Long.MIN_VALUE) { + mLastDecoderUs = buffer.decoderTimestampUs; + mLastEncoderUs = encoderUs; + } + long decoderDeltaUs = buffer.decoderTimestampUs - mLastDecoderUs; + long encoderDeltaUs = encoderUs - mLastEncoderUs; + mLastDecoderUs = buffer.decoderTimestampUs; + mLastEncoderUs = encoderUs; + long stretchUs = encoderDeltaUs - decoderDeltaUs; // microseconds that the TimeInterpolator adds (or removes). + int stretchShorts = AudioConversions.usToShorts(stretchUs, mSampleRate, mDecoderChannels); + LOG.i("process - time stretching - decoderDeltaUs:" + decoderDeltaUs + + " encoderDeltaUs:" + encoderDeltaUs + + " stretchUs:" + stretchUs + + " stretchShorts:" + stretchShorts); + processedInputSize += stretchShorts; + + // 2. Ask remixers how much space they need for the given input + processedInputSize = mRemixer.getRemixedSize(processedInputSize); + + // 3. Compare processedInputSize and outputSize. If processedInputSize > outputSize, we overflow. + // In this case, isolate the valid data. + boolean overflow = processedInputSize > outputSize; + int overflowReduction = 0; + if (overflow) { + // Compute the input size that matches this output size. + double ratio = (double) processedInputSize / inputSize; // > 1 + overflowReduction = inputSize - (int) Math.floor((double) outputSize / ratio); + LOG.w("process - overflowing! Reduction:" + overflowReduction); + buffer.decoderData.limit(buffer.decoderData.limit() - overflowReduction); + } + final int finalInputSize = buffer.decoderData.remaining(); + LOG.i("process - inputSize:" + inputSize + + " processedInputSize:" + processedInputSize + + " outputSize:" + outputSize + + " finalInputSize:" + finalInputSize); + + // 4. Do the stretching. We need a bridge buffer for its output. + ensureTempBuffer(finalInputSize + stretchShorts); + mStretcher.stretch(buffer.decoderData, mTempBuffer, mDecoderChannels); + + // 5. Do the actual remixing. + mTempBuffer.position(0); + mRemixer.remix(mTempBuffer, encoderBuffer); + + // 6. Add the bytes we have processed to the decoderTimestampUs, and restore the limit. + // We need an updated timestamp for the next cycle, since we will cycle on the same input + // buffer that has overflown. + if (overflow) { + buffer.decoderTimestampUs += AudioConversions.shortsToUs(finalInputSize, mSampleRate, mDecoderChannels); + buffer.decoderData.limit(buffer.decoderData.limit() + overflowReduction); + } + + // 7. Write the buffer. + // This is the encoder buffer: we have likely written it all, but let's use + // encoderBuffer.position() to know how much anyway. + mEncoder.queueInputBuffer(encoderBufferIndex, + 0, + encoderBuffer.position() * BYTES_PER_SHORT, + encoderUs, + 0 + ); + + return overflow; + } + + private void ensureTempBuffer(int desiredSize) { + LOG.w("ensureTempBuffer - desiredSize:" + desiredSize); + if (mTempBuffer == null || mTempBuffer.capacity() < desiredSize) { + LOG.w("ensureTempBuffer - creating new buffer."); + mTempBuffer = ByteBuffer.allocateDirect(desiredSize * BYTES_PER_SHORT) + .order(ByteOrder.nativeOrder()) + .asShortBuffer(); + } + mTempBuffer.clear(); + mTempBuffer.limit(desiredSize); + } +} diff --git a/lib/src/main/java/com/otaliastudios/transcoder/transcode/internal/VideoDecoderOutput.java b/lib/src/main/java/com/otaliastudios/transcoder/transcode/internal/VideoDecoderOutput.java index a8320fc1..da7952b1 100644 --- a/lib/src/main/java/com/otaliastudios/transcoder/transcode/internal/VideoDecoderOutput.java +++ b/lib/src/main/java/com/otaliastudios/transcoder/transcode/internal/VideoDecoderOutput.java @@ -40,8 +40,8 @@ public class VideoDecoderOutput { private EglTextureProgram mProgram; private EglDrawable mDrawable; - private final float mScaleX; - private final float mScaleY; + private float mScaleX = 1F; + private float mScaleY = 1F; private int mTextureId; private float[] mTextureTransform = new float[16]; @@ -54,15 +54,12 @@ public class VideoDecoderOutput { * Creates an VideoDecoderOutput using the current EGL context (rather than establishing a * new one). Creates a Surface that can be passed to MediaCodec.configure(). */ - public VideoDecoderOutput(float scaleX, float scaleY) { + public VideoDecoderOutput() { mScene = new EglScene(); mProgram = new EglTextureProgram(); mDrawable = new EglRect(); mTextureId = mProgram.createTexture(); - mScaleX = scaleX; - mScaleY = scaleY; - // Even if we don't access the SurfaceTexture after the constructor returns, we // still need to keep a reference to it. The Surface doesn't retain a reference // at the Java level, so if we don't either then the object can get GCed, which @@ -84,6 +81,16 @@ public void onFrameAvailable(SurfaceTexture surfaceTexture) { mSurface = new Surface(mSurfaceTexture); } + /** + * Sets the frame scale along the two axes. + * @param scaleX x scale + * @param scaleY y scale + */ + public void setScale(float scaleX, float scaleY) { + mScaleX = scaleX; + mScaleY = scaleY; + } + /** * Returns a Surface to draw onto. * @return the output surface diff --git a/lib/src/main/java/com/otaliastudios/transcoder/transcode/internal/VideoFrameDropper.java b/lib/src/main/java/com/otaliastudios/transcoder/transcode/internal/VideoFrameDropper.java new file mode 100644 index 00000000..d1c7ea85 --- /dev/null +++ b/lib/src/main/java/com/otaliastudios/transcoder/transcode/internal/VideoFrameDropper.java @@ -0,0 +1,116 @@ +package com.otaliastudios.transcoder.transcode.internal; + +import androidx.annotation.NonNull; + +import com.otaliastudios.transcoder.internal.Logger; + +/** + * Drops input frames to respect the output frame rate. + */ +public abstract class VideoFrameDropper { + + private final static String TAG = VideoFrameDropper.class.getSimpleName(); + private final static Logger LOG = new Logger(TAG); + + private VideoFrameDropper() {} + + public abstract boolean shouldRenderFrame(long presentationTimeUs); + + @NonNull + public static VideoFrameDropper newDropper(int inputFrameRate, int outputFrameRate) { + return new Dropper1(inputFrameRate, outputFrameRate); + } + + /** + * A simple and more elegant dropper. + * Reference: https://stackoverflow.com/questions/4223766/dropping-video-frames + */ + private static class Dropper1 extends VideoFrameDropper { + + private double mInFrameRateReciprocal; + private double mOutFrameRateReciprocal; + private double mFrameRateReciprocalSum; + private int mFrameCount; + + private Dropper1(int inputFrameRate, int outputFrameRate) { + mInFrameRateReciprocal = 1.0d / inputFrameRate; + mOutFrameRateReciprocal = 1.0d / outputFrameRate; + LOG.i("inFrameRateReciprocal:" + mInFrameRateReciprocal + " outFrameRateReciprocal:" + mOutFrameRateReciprocal); + } + + @Override + public boolean shouldRenderFrame(long presentationTimeUs) { + mFrameRateReciprocalSum += mInFrameRateReciprocal; + if (mFrameCount++ == 0) { + LOG.v("RENDERING (first frame) - frameRateReciprocalSum:" + mFrameRateReciprocalSum); + return true; + } else if (mFrameRateReciprocalSum > mOutFrameRateReciprocal) { + mFrameRateReciprocalSum -= mOutFrameRateReciprocal; + LOG.v("RENDERING - frameRateReciprocalSum:" + mFrameRateReciprocalSum); + return true; + } else { + LOG.v("DROPPING - frameRateReciprocalSum:" + mFrameRateReciprocalSum); + return false; + } + } + } + + /** + * The old dropper, keeping here just in case. + * Will test {@link Dropper1} and remove this soon. + */ + @SuppressWarnings("unused") + private static class Dropper2 extends VideoFrameDropper { + + // A step is defined as the microseconds between two frame. + // The average step is basically 1 / frame rate. + private float mAvgStep = 0; + private float mTargetAvgStep; + private int mRenderedSteps = -1; // frames - 1 + private long mLastRenderedUs; + private long mLastStep; + + private Dropper2(int outputFrameRate) { + mTargetAvgStep = (1F / outputFrameRate) * 1000 * 1000; + + } + + /** + * TODO improve this. as it is now, rendering a frame after dropping many, + * will not decrease avgStep but rather increase it (for this single frame; then it starts decreasing). + * This has the effect that, when a frame is rendered, the following frame is always rendered, + * because the conditions are worse then before. After this second frame things go back to normal, + * but this is terrible logic. + */ + @Override + public boolean shouldRenderFrame(long presentationTimeUs) { + if (mRenderedSteps > 0 && mAvgStep < mTargetAvgStep) { + // We are rendering too much. Drop this frame. + // Always render first 2 frames, we need them to compute the avg. + LOG.v("DROPPING - avg:" + mAvgStep + " target:" + mTargetAvgStep); + long newLastStep = presentationTimeUs - mLastRenderedUs; + float allSteps = (mAvgStep * mRenderedSteps) - mLastStep + newLastStep; + mAvgStep = allSteps / mRenderedSteps; // we didn't add a step, just increased the last + mLastStep = newLastStep; + return false; + } else { + // Render this frame, since our average step is too long or exact. + LOG.v("RENDERING - avg:" + mAvgStep + " target:" + mTargetAvgStep + " newStepCount:" + (mRenderedSteps + 1)); + if (mRenderedSteps >= 0) { + // Update the average value, since now we have mLastRenderedUs. + long step = presentationTimeUs - mLastRenderedUs; + float allSteps = (mAvgStep * mRenderedSteps) + step; + mAvgStep = allSteps / (mRenderedSteps + 1); // we added a step, so +1 + mLastStep = step; + } + // Increment both + mRenderedSteps++; + mLastRenderedUs = presentationTimeUs; + return true; + } + } + } + + + +}