WebM Movie Player (VP9/Opus) #458

Merged
merged 16 commits into from Dec 26, 2014

Projects

None yet

6 participants

@Mystler
Contributor
Mystler commented Oct 29, 2014

Now, this is what I call a collaboration.
This, finally, implements our movie player. It can play webm containers that include vp9 video and opus audio. As a bonus, it supports multiple tracks and tries to select the track that matches the game language, if possible.

@Mystler Mystler referenced this pull request in H-uru/moul-scripts Oct 29, 2014
Merged

WebM movies #89

@branan branan and 1 other commented on an outdated diff Oct 29, 2014
...ces/Plasma/FeatureLib/pfMoviePlayer/plMoviePlayer.cpp
+ SAFE_OP(fReader->Open(fMoviePath.AsString().c_str()), "open movie");
+
+ // opens the segment
+ // it contains everything you ever want to know about the movie
+ long long pos = 0;
+ mkvparser::EBMLHeader ebmlHeader;
+ ebmlHeader.Parse(fReader, pos);
+ mkvparser::Segment* seg;
+ SAFE_OP(mkvparser::Segment::CreateInstance(fReader, pos, seg), "get segment info");
+ SAFE_OP(seg->Load(), "load segment from webm");
+ fSegment.reset(seg);
+
+ // TODO: Figure out video and audio based on current language
+ // For now... just take the first one.
+ const mkvparser::Tracks* tracks = fSegment->GetTracks();
+ for (uint32_t i = 0; i < tracks->GetTracksCount(); ++i)
@branan
branan Oct 29, 2014 Member

Just so I can make sure I understand this: This will use the first audio and video tracks, UNLESS a later one happens to match the current language?

@Mystler
Mystler Oct 29, 2014 Contributor

Correct.

@branan branan and 1 other commented on an outdated diff Oct 29, 2014
...ces/Plasma/FeatureLib/pfMoviePlayer/plMoviePlayer.cpp
+
+ // open the movie with libwebm
+ fReader = new mkvparser::MkvReader;
+ SAFE_OP(fReader->Open(fMoviePath.AsString().c_str()), "open movie");
+
+ // opens the segment
+ // it contains everything you ever want to know about the movie
+ long long pos = 0;
+ mkvparser::EBMLHeader ebmlHeader;
+ ebmlHeader.Parse(fReader, pos);
+ mkvparser::Segment* seg;
+ SAFE_OP(mkvparser::Segment::CreateInstance(fReader, pos, seg), "get segment info");
+ SAFE_OP(seg->Load(), "load segment from webm");
+ fSegment.reset(seg);
+
+ // TODO: Figure out video and audio based on current language
@branan
branan Oct 29, 2014 Member

This comment is out of date

@Mystler
Mystler Oct 29, 2014 Contributor

Thanks for catching that. Updated.

@dpogue dpogue commented on an outdated diff Oct 30, 2014
...ces/Plasma/FeatureLib/pfMoviePlayer/plMoviePlayer.cpp
+ ret = x; \
+ if (ret == -1) \
+ { \
+ hsAssert(false, "failed to " err); \
+ return false; \
+ } \
+ }
+
+// =====================================================
+
+class VPX
+{
+ VPX() { }
+
+public:
+#ifdef VIDEO_AVAILABLE
@dpogue
dpogue Oct 30, 2014 Member

Should this ifdef be one line higher? Currently you'd end up with the following if VIDEO_AVAILABLE were not defined:

class VPX
{
    VPX() { }

public:
}
@Mystler
Contributor
Mystler commented Oct 30, 2014

Moved that ifdef. Also, the last libwebm release is over a year old, so I took the latest files from master and updated the TrackMgr to respect the CodecDelay (libwebm supports that now, see mkv specs). This means that our track synchronization should be a few ms more exact, now. The specific change is a one-liner squashed into 2d990ae.

Mystler added some commits Oct 26, 2014
@Mystler Mystler Make audio work
Includes some reorganization and cleanup
7cd8f51
@Mystler Mystler Calculate block alignment properly
*sigh* This happens when you copy the formula from somewhere else... Let's do it correctly now and fix the source in plVoiceChat as well.
2af4eae
@Mystler Mystler Rewrite TrackMgr
Also:
Do not start the video again if it is already playing.
Update files from libwebm
2d990ae
@Mystler Mystler Implement Pause
Also, do not show the plate before there is anything on it.
ca79d8d
@Mystler Mystler Fix video size
Scale plate using pixels and limit it to viewport size
589d59f
@Mystler Mystler Do not play if we don't know the codec 6463fad
@Mystler Mystler Choose tracks based on language, if possible 639819a
@Mystler Mystler Decode the whole audio track on startup 5563b80
@Mystler
Contributor
Mystler commented Oct 31, 2014

As I mentioned in the IRC, it looks like URU sometimes doesn't like it if we decode our audio frames on-demand. I added a new commit that changes it to preload them when we start the video. I haven't experienced any noticeable delay with this change and it fixes old record player like crackling problems in release mode.

@Hoikas Hoikas commented on the diff Nov 1, 2014
Sources/Plasma/Apps/plClient/plClient.cpp
if (!(mov->GetCmd() & plMovieMsg::kMake))
{
- for (i = 0; i < fMovies.GetCount(); i++)
+ for (i = 0; i < fMovies.size(); i++)
@Hoikas
Hoikas Nov 1, 2014 Member

It would be super awesome if this were a range-based for instead

@Mystler
Mystler Nov 1, 2014 Contributor

Uhm... i is used outside the loop, so I didn't touch it.

@Hoikas
Hoikas Nov 1, 2014 Member

Hmm... crazy code. This can be addressed later.

@Hoikas Hoikas and 1 other commented on an outdated diff Nov 1, 2014
...ces/Plasma/FeatureLib/pfMoviePlayer/plMoviePlayer.cpp
@@ -42,25 +42,417 @@ You can contact Cyan Worlds, Inc. by email legal@cyan.com
#include "plMoviePlayer.h"
+#ifdef VIDEO_AVAILABLE
@Hoikas
Hoikas Nov 1, 2014 Member

This is kind of ambiguous--are we referring to vpx+opus or just vpx?

@Mystler
Mystler Nov 1, 2014 Contributor

In your original version it was VPX_AVAILABLE which I found misleading. VIDEO should represent both libs, but I can change it to MOVIE_AVAILABLE, if you prefer that.

@Hoikas Hoikas and 1 other commented on an outdated diff Nov 1, 2014
...ces/Plasma/FeatureLib/pfMoviePlayer/plMoviePlayer.cpp
+ hsAssert(false, "failed to " err); \
+ return false; \
+ } \
+ }
+
+// =====================================================
+
+class VPX
+{
+ VPX() { }
+
+#ifdef VIDEO_AVAILABLE
+public:
+ vpx_codec_ctx_t codec;
+
+ ~VPX() {
@Hoikas
Hoikas Nov 1, 2014 Member

code style nitpick

@Mystler
Mystler Nov 1, 2014 Contributor

Guess who wrote that. :p

@Hoikas Hoikas and 1 other commented on an outdated diff Nov 1, 2014
...ces/Plasma/FeatureLib/pfMoviePlayer/plMoviePlayer.cpp
+ if (!fCurrentBlock)
+ fStatus = fTrack->GetFirst(fCurrentBlock);
+
+ // Continue through the blocks until our current movie time
+ while (fCurrentBlock && fStatus == 0)
+ {
+ const mkvparser::Block* block = fCurrentBlock->GetBlock();
+ int64_t time = block->GetTime(fCurrentBlock->GetCluster()) - fTrack->GetCodecDelay();
+ if (time <= movieTimeNs)
+ {
+ // We want to play this block, add it to the frames buffer
+ frames.reserve(frames.size() + block->GetFrameCount());
+ for (int32_t i = 0; i < block->GetFrameCount(); i++)
+ {
+ const mkvparser::Block::Frame data = block->GetFrame(i);
+ uint8_t* buf = new uint8_t[data.len];
@Hoikas
Hoikas Nov 1, 2014 Member

potential optimization: move the allocation step out of the loop and only (re-)allocate if the existing buffer is not big enough

@Mystler
Mystler Nov 1, 2014 Contributor

I could use a std::vector to do that, I assume.

@Mystler
Mystler Nov 1, 2014 Contributor

Actually, I'm not sure how to approach this, since that buffer is going into the frames container as a unique pointer.

@Hoikas
Hoikas Nov 1, 2014 Member

Oh, I thought this was a buffer that did not have a long life span. Whoops!

@Hoikas Hoikas commented on an outdated diff Nov 1, 2014
...ces/Plasma/FeatureLib/pfMoviePlayer/plMoviePlayer.cpp
+ hsAssert(false, "Not an Opus audio track!");
+ return false;
+ }
+ int error;
+ OpusDecoder* opus = opus_decoder_create(48000, audio->GetChannels(), &error);
+ if (error != OPUS_OK)
+ hsAssert(false, "Error occured initalizing opus");
+
+ // Decode audio track
+ std::vector<blkbuf_t> frames;
+ fAudioTrack->GetFrames(fReader, fSegment->GetDuration(), frames);
+ static const int maxFrameSize = 5760; // for max packet duration at 48kHz
+ std::vector<int16_t> decoded;
+ decoded.reserve(frames.size() * audio->GetChannels() * maxFrameSize);
+
+ for (auto it = frames.begin(); it != frames.end(); ++it)
@Hoikas
Hoikas Nov 1, 2014 Member

range based for would be super awesome

@Hoikas Hoikas commented on an outdated diff Nov 1, 2014
...ces/Plasma/FeatureLib/pfMoviePlayer/plMoviePlayer.cpp
+ if (error != OPUS_OK)
+ hsAssert(false, "Error occured initalizing opus");
+
+ // Decode audio track
+ std::vector<blkbuf_t> frames;
+ fAudioTrack->GetFrames(fReader, fSegment->GetDuration(), frames);
+ static const int maxFrameSize = 5760; // for max packet duration at 48kHz
+ std::vector<int16_t> decoded;
+ decoded.reserve(frames.size() * audio->GetChannels() * maxFrameSize);
+
+ for (auto it = frames.begin(); it != frames.end(); ++it)
+ {
+ const std::unique_ptr<uint8_t>& buf = std::get<0>(*it);
+ int32_t size = std::get<1>(*it);
+
+ int16_t* pcm = new int16_t[maxFrameSize * audio->GetChannels()];
@Hoikas
Hoikas Nov 1, 2014 Member

we could optimize this by moving the allocation outside of the loop. (heap alloc is slow)

@Hoikas Hoikas commented on an outdated diff Nov 1, 2014
...ces/Plasma/FeatureLib/pfMoviePlayer/plMoviePlayer.cpp
{
- return ((int64_t) hsTimer::GetSeconds() * fTimeScale) - fStartTime;
+ auto codes = plLocalization::GetLanguageCodes(plLocalization::GetLanguage());
+ if (codes.find(track->GetLanguage()) != codes.end())
+ return true;
+ return false;
+}
+
+void plMoviePlayer::IProcessVideoFrame(const std::vector<blkbuf_t>& frames)
+{
+#ifdef VIDEO_AVAILABLE
+ vpx_image_t* img = nullptr;
+
+ // We have to decode all the frames, but we only want to display the most recent one to the user.
+ for (auto it = frames.begin(); it != frames.end(); ++it)
@Hoikas
Hoikas Nov 1, 2014 Member

range based for?

@Hoikas Hoikas and 1 other commented on an outdated diff Nov 1, 2014
...ces/Plasma/FeatureLib/pfMoviePlayer/plMoviePlayer.cpp
+
+bool plMoviePlayer::Start()
+{
+ if (fPlaying)
+ return false;
+
+#ifdef VIDEO_AVAILABLE
+ if (!IOpenMovie())
+ return false;
+ hsAssert(fVideoTrack, "nil video track -- expect bad things to happen!");
+
+ // Initialize VPX
+ const mkvparser::VideoTrack* video = static_cast<const mkvparser::VideoTrack*>(fVideoTrack->GetTrack());
+ if (strcmp(video->GetCodecId(), WEBM_CODECID_VP9) != 0)
+ {
+ hsAssert(false, "Not a VP9 video track!");
@Hoikas
Hoikas Nov 1, 2014 Member

this should probably be a log message--people who embed videos in ages they are creating probably won't be using a debug client. therefore, they will not see this assert

@Mystler
Mystler Nov 1, 2014 Contributor

I assume the same goes for the audio track? Where would you recommend to log that message to?

EDIT: In my WIP, I replaced the hsAssert with a log entry to a movie.log file.

@Hoikas
Hoikas Nov 1, 2014 Member

That sounds good

@Hoikas Hoikas commented on an outdated diff Nov 1, 2014
...ces/Plasma/FeatureLib/pfMoviePlayer/plMoviePlayer.cpp
}
bool plMoviePlayer::Stop()
{
- for (int i = 0; i < fCallbacks.GetCount(); i++)
+ fPlaying = false;
+ if (fAudioSound)
+ fAudioSound->Stop();
+ if (fPlate)
+ fPlate->SetVisible(false);
+
+ for (int i = 0; i < fCallbacks.size(); i++)
@Hoikas
Hoikas Nov 1, 2014 Member

range based for?

@Hoikas
Member
Hoikas commented Nov 1, 2014

I know I wrote the original code here, and it's probably awful that I'm going to say this, but I've since changed my mind about how our code formatting should look. It would be really awesome if we could get this to use the same style as DirtSand. My newer additions, such as pfPatcher sort of do this. The most relevant change here is that class and method definitions have a newline before the opening braces, but loops and conditional statements do not.

Also, I believe congratulations are in order for getting this to work!

@Mystler
Contributor
Mystler commented Nov 1, 2014

Heh, I definitely agree. The Plasma code style is really annoying. In fact, when writing on this I sometimes had to fight with my VS editor, because it didn't like how I wrote. Neither did I.

If you say that it's time for a revolution, I'm going to jump right into the front lines.

@Deledrius
Member

It can be hard to decide how to write any given thing, since there's no consistent style through the entire codebase (which given its size and the length of time it was worked on, isn't surprising), or sometimes even in within the same project.

So, is it finally time to merge this: 8590ba4?

@Mystler
Contributor
Mystler commented Nov 1, 2014

Updated. One commit addresses what you pointed out, the other one changes the code style.

@Hoikas
Owner

Becaust const and because reference. ;)

@Lunanne
Member
Lunanne commented Nov 26, 2014

Tested it and seems to work fine.
The only issue I found is that when you pres the escape key more than once to exit the intro video (because you're impatient and there is a slight delay) you can get the help gui to show up more than once. You can just click them away so I don't think this is a huge issue. The screen did flicker for me sometimes during the intro video, but that might be on my end.

@Deledrius
Member

👍

@Hoikas Hoikas merged commit 990f42c into H-uru:master Dec 26, 2014
@Mystler Mystler deleted the Mystler:webm branch Dec 26, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment