Train new model

[Agamemnus-uA]

Hi, I've been on a mission to benchmark different encoders and their settings using VMAF as well as some other metrics. I get 2 main points of feedback. 1, which I think is incorrect, is that you can't use model 0.6.1 on a reference/distorted pair in 720p. I believe most people are actually incorrect in this sense, since I don't compare absolute scores across resolutions, I only compare 720p reference/distorted with other distorted 720p encodes of the same source, and see which one performs higher. Same goes for 1080p and 1440p, all distorted encodes are the same resolution as the reference in my benchmarks. Can you confirm this is fine or not?

The second main point of feedback I get is that VMAF isn't tuned to predict gameplay footage. Since I'm focused on encoding choices/options for gameplay footage, I can't help but shake the feeling that the dataset you've used to train the models you offer is in fact not suited to this. Can you confirm this is true, to either a small or large degree?

If so, I'm very interested in training a new model but I have no idea how to do it and would need help. I would think something like 1080p at 2xH viewing distance or maybe 1440p at 1.5xH viewing distance might be an appropriate underlying assumption for most computer/desk viewer situations. I have access to hundreds of people I can use to gather opinion score data, with a variety of distortion versions from an original, and can do it across multiple references (say 3 or 4 different games). The thing is, I have no idea how to store these scores in a way that a training mechanism like libsvm will read, and I have no idea how to generate a final model file that I can then actually use through FFMPEG and the libvmaf filter.

If this idea is a waste of time, and there's little reason to create a new model for gameplay footage for desk/pc viewers, then I'd love to have that confirmed here. But if there's value in doing this, I'm very interested in being a part of it, but I'd essentially need a n00b tutorial on how to create the model. I will happily share any and all opinion and benchmark data you like with you here or elsewhere.

If you want to see some of the benchmarks I've been doing here's a few recent ones:
https://unrealaussies.com/tech/x264-presets-and-settings-quality-analysis-of-720p60-apex-legends/
https://unrealaussies.com/tech/nvenc-x264-quicksync-qsv-vp9-av1/
https://unrealaussies.com/tech/nvenc-x264-obs/

[slhck]

Can you confirm this is fine or not?

Note: I am not affiliated with this project.

You can find some background info here: https://github.com/Netflix/vmaf/blob/master/resource/doc/models.md#predict-quality-on-a-1080p-hdtv-screen-at-3h – what that means is that in principle you can compare ref/dist videos at other resolutions, but you would have to scale the (assumed) viewing distance accordingly.

I would think something like 1080p at 2xH viewing distance or maybe 1440p at 1.5xH viewing distance might be an appropriate underlying assumption for most computer/desk viewer situations

As for appropriate viewing distances, see ITU-R BT.500-13 and ITU-R BT.2022. Typically, for 1080p, one would choose 3H viewing distance, and 1.5–1.6H for 2160p.

I have access to hundreds of people I can use to gather opinion score data, with a variety of distortion versions from an original, and can do it across multiple references (say 3 or 4 different games)

Note that when training a model for one particular type of content (e.g., just gaming videos), it will be biased towards that and cannot be used for other content. If you're fine with that, I would recommend choosing at least 4–6 different reference videos (SRCs). Make sure that they span a broad range of spatiotemporal complexity (you can use SI/TI to measure that), or encode the reference with a fixed CRF level and check the resulting bitrate to get an estimate of its complexity.

You also won't need hundreds of people; typically, 25–30 is enough for statistical purposes. In a laboratory setting, that is. If you're talking about crowdsourcing / people watching at home, that's a different story.

If you can do more than one subjective test (i.e., spread your selection of test clips across multiple panels of viewers who each rate only a fraction of the clips), you could align the subjective tests later (e.g., such as shown here).

Also, you might be interested in a current ITU-T work item on subjective evaluation of gaming quality; see ITU-T Rec. P.809 as well as these publications:

• https://ieeexplore.ieee.org/abstract/document/8463403
• https://ieeexplore.ieee.org/document/8463362

[Agamemnus-uA]

Yeah to compare absolute scores of a 720p scaled version of a reference 1080p clip with other 1080p distorted versions, you would first need to upscale the distorted 720p. I'm totally on board with that.

I only suggest a different viewing distance because the dataset would be target to typical Twitch or YouTube stream viewing on a PC. My impression is that those recommendations are targeted at either lounge-room viewing, or judging actual display hardware quality. I'm happy to go with those anyway. I just figured getting people to watch it the same way they'd normally watch Twitch or YouTube gaming on would be real-world representative of the actual use. The purpose is exactly what you say next, with the model being for a particular type of content. That's one of the feedback points I often get for using model 0.6.1 since it's made for Netflix content, instead of common game streamer content. This is the reason for me asking about how to train a new model, and whether or not it's worth it, or is model 0.6.1 already sufficient enough for that kind of content.

Obviously only getting 30 people will be easier, but I'm more than happy to get hundreds or even thousands of subjective tests done just to increase the confidence of the prediction. Also happy to use multiple panels on sample sets, in fact, I might HAVE to do it that way.

It kind of boils down to 2 main problems:
1 - Is there any point? Is 0.6.1 already more than qualified to reliably assess content such as Overwatch or League of Legends gameplay? Or would this be of measurable benefit?
2 - How to actually do it. I have no idea how to supply the subjective results to the learning mechanism, nor how to even run it. This is the main thing I would need help with.

[slhck]

I just figured getting people to watch it the same way they'd normally watch Twitch or YouTube gaming on would be real-world representative of the actual use.

Good point. In this case, you could indeed use a lower viewing distance. Note that the suggested 1.5 H for UHD video is still, in my opinion, far from a "living room" or "lounge viewing" scenario. It's just the critical distance for being able to even see visual differences.

Is there any point? Is 0.6.1 already more than qualified to reliably assess content such as Overwatch or League of Legends gameplay? Or would this be of measurable benefit?

Personally, I think you would benefit from creating a model for gaming video if you have the resources to do that.

How to actually do it. I have no idea how to supply the subjective results to the learning mechanism, nor how to even run it. This is the main thing I would need help with.

See https://github.com/Netflix/vmaf/blob/master/resource/doc/VMAF_Python_library.md#advanced-usage

[Agamemnus-uA]

Not sure how I never found that guide before. I'll give it a whirl in about 2 weeks for a test run.

Is there a way to do this in parts? That is, say I collect 20 opinions on 2 distortions per each of 3 references, I test building a model and it works... THEN I add another 3 references each with say 5 distortions and get about 50 opinions on each, and then UPDATE the existing model?

The idea being, that it can be tested, then trained on 2 different games as source footage, then expanding to include a 3rd game, 4th game, then a 5th game, and start there with a v1.0 model file. Then update that model file to be v1.1 by adding another 50 opinions on new distortions, then another 50 opinions on new distortions for a v1.2 so that eventually, it's incorporating distortions from more and more encoders and variations of each encoder.

The plan I have in mind, is to do it with some x264 and x265 encodes, then to do it again with libx-vp9, NVENC, QuickSync, then possibly again with something like SVT-HEVC, SVT-VP9 and finally libaom-av1 and SVT-AV1. Upgrading it to always include the distortions from newer encoders seems like it makes sense. But if I have to train it from scratch each time, then so be it.

[slhck]

You could certainly add more files and subjective scores to a dataset file later on, and train another model, but from a test design perspective this is, in my opinion, a no-go.

There's an interesting paper by Zielinski et al. that describes how human subjects cannot give absolute ratings of quality, and how they will adjust their ratings to the presented stimuli.

What you'll find is that if your individual tests do not cover the same range of absolute quality (on an imaginative absolute quality scale) and the same kind of content (and I guess that different games will be judged very differently), you risk obtaining biased subjective ratings, which you cannot just pretend to come from the same study.

For example, if one test contains a lot of H.264 blocking artifacts for high-motion content (e.g., a first person shooter), and another only HEVC-encoded role playing game footage, your quality scores might still cover a range from 1–5, since the definition of "worst" and "best" depends on what you've shown to the subjects. But if directly compared the worst clip in both tests, you'd find that they'd now achieve different subjective scores.

Now, this bias may be less present with an impairment scale (DMOS), but I'm curious to see what @li-zhi and others think about that, and how to perform "incremental" training.

[Agamemnus-uA]

I think that's a good point actually.

With regards to subjective testing, do I just get them to rank it on a scale of 1-5 like in that paper? I understand why 1 was mapped to 20 and 5 mapped to 100. Do I just fill out the dataset file with 'mos':100, instead of dmos? I don't fully understand the difference between dmos and mos nor how to calculate them.

[slhck]

In order to understand what DMOS is and how it differs from MOS, you'd have to read the relevant ITU standards. I once tried to summarize the most widely used methods here, including references: https://en.wikipedia.org/wiki/Subjective_video_quality#Standardized_testing_methods

• If you choose an absolute category rating (ACR) approach where people rate individual clips (without knowing what the reference is) on a scale from "bad" to "excellent", you get a MOS.

• If on the other hand you always show people two clips (the reference and a distorted video) and ask them to judge the differences ("very annoying" to "imperceptible"). That would be called DSIS or DCR.

• There's a hybrid variant where you do an ACR test but include an unimpaired reference, however, subjects don't know about it. This is ACR-HR.

Generally, DSIS tests provide more fine-grained ratings (i.e., people are able to judge smaller differences than in an ACR test when shown the same material), so that'd be the most useful method for training the model you're trying to develop. But ACR is also fine, especially if it's more about having people watch clips "naturally" and without explicitly looking for distortions.

I should add that performing valid and reliable subjective tests is really not a simple task.

Oh, and if I haven't linked to that yet, you may be interested in: https://www.researchgate.net/publication/325285444_An_Evaluation_of_Video_Quality_Assessment_Metrics_for_Passive_Gaming_Video_Streaming

[Agamemnus-uA]

OK great that's a good summary. Obviously the easiest is ACR, and although I suspect DSIS would be useful in many circumstances, I can see what you're saying about how "natural viewing" might mean that it provides no significant advantage for the exact purpose I have in mind.

I've requested that paper, I do have Institutional access to ResearchGate but I still needed to file a specific request. It would be a great place to start to know if my model is succeeding by comparing it to the model that they used SPECIFICALLY on game footage. Although, in my OP I was wondering about VMAF's reliability in gameplay-specific footage, this paper appears to answer that one fairly decisively, I just need to confirm the model that they used in it.

You've been extremely helpful and thank you for your time.

I'll get back to you in a week or two when I finish some assignments and get to testing the training procedure on some test data. Might have more specific code questions then. But while I'm here, do you know if there's any way in FFMPEG to use libvmaf filter to produce multiple pooling results? Say the aggregate default + harmonic mean + perc10 or something? I have thousands of distorteds to run through and I'd rather not run it multiple times on each. In fact, I already have thousands of XML logs for the default pooling method, is there already a convenient way to then use those logs which contain per-frame scores to calculate the results again just with another pooling method?

[slhck]

You may also be interested in this other issue that was just opened: #321

It would be a great place to start to know if my model is succeeding by comparing it to the model that they used SPECIFICALLY on game footage

I guess it will be important to create different training and test sets even when building your model to check its performance. I can also get you in touch with the authors — I know all of them personally, and I believe they'd be interested in learning about what other people are doing.

do you know if there's any way in FFMPEG to use libvmaf filter to produce multiple pooling results?

Not to my knowledge. I'd probably write a wrapper to do these calculations (pooling) in Python using the original scores.

[Agamemnus-uA]

Poor guy, I suspect it's to do with how his capture card captures the video, but need more info.

They did get back to me and send me the full journal article and it's beautiful. They specify the model and how they correlated the scores. Essentially, the idea would be that a game-specific model will correlate better, it's a little ambitious but I'm happy to try. If it works out it works out. I'm pretty booked for another 6-7 days but I'll start it up after that.

What you say about training and testing sets is good, I've been thinking about it. Once I've confirmed I can successfully train a model at all, I'll develop a test set and split the results. I'll only train on the results from say 70% of the opinions from 70% of the games. Then do a verification of the trained model against the remaining scores, some of which will be games it was originally trained on, but some of the games will be ones it was NOT trained on. I might even split the verification into 3 parts, one part is verified against the opinions it was NOT trained on for the games that it HAS seen, one for the games it has never seen, and one which is a combination of both. Doing this might reveal whether or not it would benefit from more games being included in the training, and help decide whether to start the process again or not.

That said, I've got my heart set on ACR-HR and hiding the original in with the test set. I'm thinking around 6 games, 2 clips per game, 8 distortions per clip for 96 distortions plus 12 references total. What are your thoughts? Do you think it matters if some viewers view only some of the ref/distortion sets? It might pay to increase the test size if this is fine.

[slhck]

The numbers sound good if your clips are ~10 s long. You don't want your test to be longer than 1 hr, including breaks. Rating quality is really exhausting.

Have a look at https://vqeg.github.io/software-tools/tools/#subjective-test-software if you need some tools for running these tests.

It does not matter much if not all subjects see all distorted clips. But you should try to show all games (SRCs) to everyone, and everyone should see at least a subset of the distortions (HRCs) paired with the SRCs which exhibit the full quality range.

In other words, if you plan on running a series of subjective studies with different viewers and different content, there should be an overlap (“common set”) based on which you can later merge the scores into one big dataset (various procedures for doing so exist). Your common set should include both the lowest and highest expected quality, and one or two clips in the middle of the quality range.

[Agamemnus-uA]

Yeah makes sense.

Do you know if ./run_testing and ./run_vmaf_training will work if my dataset has 'mos' instead of 'dmos' like in NFLX_dataset_public.py ? While thinking about how to get users the files, it turns out that inserting a single lossless (reference) version to ascertain the DMOS will severely limit the audience because of the data size, not to mention I'll have to disguise ALL OF THE distorted versions in a lossless format so that they can't cheat, it will be too much data for most people and my sample size will be chopped into a fraction.

Better yet, will it accept 'dmos' on SOME content IDs, but 'mos' on other content IDs? That way I can incorporate both methods into a dataset, using live tests for DMOS and remote tests for MOS ?

[slhck]

Ah, I see. Are you intending to have users run the test at home, in a non-standardized environment, using their own equipment and software, potentially even manually starting the videos? Be aware that this kind of “crowdsourcing” is going to create a lot of noise in the data that may lead to biased or unreliable training results. You cannot control viewing distance, lighting, display equipment, the amount of distraction, errors in the playout, …
In this case, you will need a lot more users (I'd say around 150–250 instead of 25 per test) and at least a rating software that makes it impossible for users to cheat and know what they're watching.

See also:

• https://ieeexplore.ieee.org/abstract/document/6671455/
• https://www.researchgate.net/profile/Philippe_Hanhart2/publication/263010491_Survey_of_Web-based_Crowdsourcing_Frameworks_for_Subjective_Quality_Assessment/links/00b7d53a16021c0da8000000/Survey-of-Web-based-Crowdsourcing-Frameworks-for-Subjective-Quality-Assessment.pdf

I don't know about the specifics of the training functions – best to wait for someone from Netflix to help out.

[Agamemnus-uA]

OK Hopefully one of them spots that question.

And yes, although I could do a controlled test at work, it would have controlled parameters. This comes back to the whole "how you would normally watch a Twitch or YouTube stream" concept. First difference, it's almost always video gameplay instead of the typical Netflix kind of source. Second difference is that users will have many different setups. The sole attempt of this endeavour would be to capture these factors into the prediction mechanism. Ideally, the end result model would be a prediction of how viewers on average would rate a stream's quality, and the average viewer in such a case, is by it's nature, noisy.

Currently, if you take a look around the net, there's thousands of different claims about these encoder settings being better than those encoder settings at such and such bitrate, but there's very VERY little proof. Pull out some VMAF scores and the first things people say are "that's not tuned to gameplay" and "It doesn't work if you have a different res, or sit a different distance from your monitor, or view from a different angle etc etc etc" I'll take as many samples as I need as long as they continue to improve the prediction accuracy until I'm sure either it won't get better, or it won't be able to beat the 0.6.1 model. I realise you can't know what a person will think, because essentially, people disagree all the time. The point will be to say that "93% of people will agree that A is better than B" and actually be able to back it up with some kind of facts. The journal piece you linked earlier is actually perfect in terms of evidence/proof, but they used a default model, I'm only left wondering if there can be a better one.

[li-zhi]

Hi there, just wanted to add a few points after reading thorough this thread (hopefully they provide a some complements to @slhck's suggestions):

1. It is fine to use model 0.6.1 to benchmark 720p videos (with both source and encode at 720p without upscaling), as long as you interpret the absolute scores with viewing distance 4.5H. And if you only use the relative scores to benchmark codecs, there is no problem either.

2. The (perhaps) limitation of VMAF when applied to gaming videos, as I learned from several conversations, is that, the score got boosted with fast head movements (e.g. in 360 videos, or first-person shooting games). This is due to the fact that VMAF is trying to capture the temporal masking effect. In the context with gaming videos, one often observes high scores associated with, say, fast camera panning. The correct interpretation of this is that "the subject wouldn't be able to notice the coding or other artifacts during fast movement", but perhaps this makes VMAF less useful? It would be really great to hear from the gaming community how to modify VMAF better for their use cases.

3. Regarding training VMAF at a viewing distance other than 3H: the viewing distance's effect should be captured in two different aspects: 1) the SVM model, when trained to align predictions with subjective scores, 2) in ADM (DLM) feature, there is a viewing distance-specific parameter, see here:
   https://github.com/Netflix/vmaf/blob/master/feature/src/adm_tools.h#L275
   You may want to adjust this parameter according to your subjective test's viewing distance.