Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FR: Loudness scanning and normalization - new requirements #1046

Open
AironAudio opened this issue Sep 21, 2018 · 29 comments
Open

FR: Loudness scanning and normalization - new requirements #1046

AironAudio opened this issue Sep 21, 2018 · 29 comments

Comments

@AironAudio
Copy link

@AironAudio AironAudio commented Sep 21, 2018

Netflix has just changed its delivery requirements. We now need to use a Dolby dialogue-gated(which is now free code), 1770.1 version of the loudness measurement standard. That's the 1770 standard without a relative gate.

http://www.itu.int/dms_pubrec/itu-r/rec/bs/R-REC-BS.1770-0-200607-S!!PDF-E.pdf

The libebur128 library has been updated as well, and would need to be modified to bypass the relative and absolute gates.

Why Reaper ?
Speed. I normalize the items of the 5.1 or 2.0 mixes in Reaper with the SWS extension and do the split-to-mono playouts that I need to pack up and send off. With other tools there's at least one more inbetween step, as none of them seem to be able to generate single-channel files. Thus, I'd rather use Reaper, even if I need to get another tool in the meantime. Reaper's just the better tool.

Currently the Loudness portion of the extension uses the BS-1770.3 or 4 method of measuring as well as the EBU R128 recommendation.

The whole thing is based on this article which links to the actual recommendations:
https://www.pro-tools-expert.com/home-page/2018/9/18/netflix-respond-to-our-article-on-their-new-loudness-delivery-specs

Basically it's a dialogue gate whose output is scanned by the ITU 1770-1 method. We're using the pure 1770-3 method irrc, or perhaps even 1770.4.

The code for the Dolby dialogue gate is completely free. It's available on request via this link : https://www.dolby.com/us/en/technologies/speech-gating-reference-code.aspx

The Netflix recommendation is here: https://help.prodicle.com/hc/en-us/articles/360001759947-Netflix-Audio-Mix-Specifications-and-Best-Practices

Reaper's extension architecture just lends itself so well to offline scanning, that I'm hoping there's a programmer with a spare few hours to slot in an extra mode in to the existing Loudness portion of the SWS extension.

I've got no C++ IDE experience at all, or I'd try to do this. I'd donate a good chunk to get this in to the SWS extension. The tools to do it more slowly than in Reaper cost at least $400, so I'd be perfectly willing to drop at least $200 in donations.

@AironAudio AironAudio changed the title Loudness scanning and normalization - new requirements FR: Loudness scanning and normalization - new requirements Sep 21, 2018
@AironAudio
Copy link
Author

@AironAudio AironAudio commented Sep 22, 2018

After filling out the form on the Dolby site, you can download the code right away. Nobody sits there approving any organization or person.

Checking the "Dolby Dialogue Intelligence" code, it's very well documented pure ANSI C. It converts the input to 16kHz, runs it through a bunch of detectors and combines the results to get a value on whether speech is there or not. That probably has to be used to control a gate or volume parameter across the sections.

Still studying the stuff.

@nofishonfriday
Copy link
Collaborator

@nofishonfriday nofishonfriday commented Sep 26, 2018

Just took a look at the Dolby Dialogue Intelligence source files, it says:

This program is protected under international and U.S. copyright laws as an unpublished work. This program is confidential and proprietary to Dolby Laboratories. Reproduction or disclosure, in whole or in part, or the production of derivative works therefrom without the express permission of Dolby Laboratories is prohibited.

Someone may correct me, but I don't think it's possible to incorporate this in SWS, being open source / MIT licensed, is it ?

@AironAudio
Copy link
Author

@AironAudio AironAudio commented Sep 26, 2018

That is correct. For that reason I may need to make an extension all by itself for which the source code can only be made available who have gotten the source from Dolby itself, which is free and only requires filling out that form.

I'm investigating how to do that. I've got Visual Studio installed, got some extension code form Cfillion to start with and the all the other source code, but to be honest, that could take a very long time, because my experience level with C++ is almost zero.

@cfillion
Copy link
Collaborator

@cfillion cfillion commented Oct 1, 2018

The foreign (dolby's) source code does not need to be integrated into the open-source codebase. Each interested developer could download their own copy and have the open-source part pick it up at build time (this could somewhat complicate things for automated CI builds which seems to be on the table however.)

Another possibility is to make the dolby code available to SWS's existing loudness code via a DLL (assuming supporting that and 1770.1 doesn't require rewriting the whole thing... in which case re-inventing the wheel from scratch in a new extension might be easier).

@AironAudio
Copy link
Author

@AironAudio AironAudio commented Oct 1, 2018

The DLL approach seems feasible. Thank you. If I can figure out how to make a DLL of that ANSI C code, perhaps someone at SWS can create a modified copy of the existing loudness code from the libebur128 library. The Dolby code is supposed to act as an on/off gate only.

Scan sample block with dialogue gate, gate sample block, scan with 1770-1 loudness scan. That 1770-1 version is just the same without the relative gate in the 1770-3/4 version.

Learning C++ atm, so I don't know how long this might take. Working a lot at the same time too. I'll be asking questions :) .

@nofishonfriday
Copy link
Collaborator

@nofishonfriday nofishonfriday commented Oct 4, 2018

@AironAudio
I'm currently in the process of updating the in SWS used libebur128 library to latest version (v1.2.4), something I had in mind to do for longer anyway.
First runs with the EBU provided testfiles look ok so far.
Here's a SWS test version with the updated library:
https://www.dropbox.com/s/kv2383b2tawjiai/reaper_sws64.dll?dl=1

Branch is here:
https://github.com/nofishonfriday/sws/tree/libebur128_-_update_to_v1.2.4

Would be nice if you (or anyone interested) could give it a test drive and compare results with other Loudness scanners if you have a chance. I only have the free r128gain available. Results also seem ok there so far, but it doesn't show short term / momentary max.

Regarding implementing the Netflix loudness scanning in SWS:
An idea would be doing a FR issue for implementing functions to bypass the gating at https://github.com/jiixyj/libebur128 directly, maybe they'll also find it useful.

This would save us from doing the mod ourselves

@AironAudio
Copy link
Author

@AironAudio AironAudio commented Oct 11, 2018

Finally found time to do a couple of scans. I'm currently evaluating another application, Nugen LM Correct.

The 1770-4 scans from that matches the updated library you posted for integrated loudness. The maximum short term (3 second loudness) and maximum momentary does vary a bit. LM Correct gets higher values quite consistently, so they must be measuring stuff a little differently.

The files that have to check out are the test files. I got them here:
https://tech.ebu.ch/publications/ebu_loudness_test_set

And here's a spread sheet of exported stuff from the Loudness window for all those files.
https://www.dropbox.com/s/b548kfxwolwy1cu/SWS%20-%20libebur128%20v142%20-%201770-4%20scans.ods?dl=0

What these values need to be is detailed here: https://tech.ebu.ch/docs/tech/tech3341.pdf on page 10+

and here https://tech.ebu.ch/docs/tech/tech3342.pdf

They all check out for "file based" scanners as far as I can tell, except for the seq-3341-13-*.wav files. Those are the only ones that do not check out, so there's something up with the momentary calculations. Only the first file, seq-3341-13-1-24bit.wav check out at -23 LUFS maximum momentary loudness.

This is something we need to post for the libebur128 guys too.

Other than that, it's all great. Integrated loudness over entire mixes all check out file, as do all the true-peak measurements, which I never use however, since my oversampled limiter takes care of those.

@nofishonfriday
Copy link
Collaborator

@nofishonfriday nofishonfriday commented Oct 11, 2018

That's helpful, thanks.

I just did a loudness scan with the EBU testfiles with the libebur v1.0.1 that's used in SWS currently and I get the exact same results (also short term / momentary max.) as with my updated v1.2.4 version, so I'm not even sure if it's even worth updating the lib (seems none of the fixes affect us). The lib is quite modified for use in SWS so the updating was not that straight forward and may have introduced new issues during my update process.
I'll probably submit it to SWS as optional pull request requesting a review.

@nofishonfriday
Copy link
Collaborator

@nofishonfriday nofishonfriday commented Oct 11, 2018

Posted at libebur128 about the seq-3341-13-* files, seems the issue is on our end.
jiixyj/libebur128#93

Will try to look into it.

@audionuma
Copy link

@audionuma audionuma commented Oct 11, 2018

Concerning max momentary issue, see my comment on the libebur128 closed issue:
jiixyj/libebur128#93 (comment)

@nofishonfriday
Copy link
Collaborator

@nofishonfriday nofishonfriday commented Oct 12, 2018

Found the culprit, thanks to @audionuma's hint.

loudn2

@BlackScreen
Copy link

@BlackScreen BlackScreen commented Oct 12, 2018

@nofishonfriday: When can we use it in SWS? I had also noticed this when testing with the EBU files.

@AironAudio
Copy link
Author

@AironAudio AironAudio commented Oct 12, 2018

Nice one. With that we have 1770-4 compliance.

Next thing on my bucket list is trying to make a DLL out of the Dolby Dialogue Gate code. I've got most of the stuff installed and tutorials at hand, because I realize it's an extremely niche thing I need.

I'm using the Nugen LM Correct tool atm, and it's a bit slow. You can't turn off truepeak scanning, and I only analyze files and do all the gain changes in Reaper anyway, because that's where I render to multiple mono files with the limiter of my choice, and across as many episodes as I need. The Nugen tool is one-at-a-time and has no drag'n'drop. No commandline version either.

That's my motivation to get the Dolby Dialogue Intelligence Gate in to the SWS extension. I'm hoping it won't take terribly long. Breeder is terribly busy.

I will ask questions once I get that library compiled somehow. Maybe even create a little extension. Who knows.

nofishonfriday added a commit to nofishonfriday/sws that referenced this issue Oct 17, 2018
- Revert changing relative gating threshold from -10.0 LU to -20.0 LU
- add 'high precision' analyzing mode (see reaper-oss#1046)
@nofishonfriday
Copy link
Collaborator

@nofishonfriday nofishonfriday commented Oct 17, 2018

I've added a new 'high precision' analyzing mode which should give correct results with the seq-3341-13-* files. Can be set in 'SWS/BR: Analyze loudness...' window -> Options' or via new action 'SWS/BR/NF: Toggle use high precision mode for loudness analyzing'.

Test version here (Win x64)
Branch here

nofishonfriday added a commit to nofishonfriday/sws that referenced this issue Oct 17, 2018
- Revert changing relative gating threshold from -10.0 LU to -20.0 LU
nofishonfriday added a commit to nofishonfriday/sws that referenced this issue Oct 17, 2018
…ance

see reaper-oss#1046 (comment)
- Revert changing relative gating threshold from -10.0 LU to -20.0 LU
nofishonfriday added a commit to nofishonfriday/sws that referenced this issue Oct 17, 2018
…ance

see reaper-oss#1046 (comment)
- Revert changing relative gating threshold from -10.0 LU to -20.0 LU
@AironAudio
Copy link
Author

@AironAudio AironAudio commented Oct 17, 2018

Nice. Does that mean it may be easier adding other scan modes in the future as well ? I'll give the build a test run here too.
I'm finishing up a bunch of mixes in the coming days and will try to get some coding done in the week or so of free time.

@nofishonfriday
Copy link
Collaborator

@nofishonfriday nofishonfriday commented Oct 17, 2018

Thanks if you get a chance for testing.

Does that mean it may be easier adding other scan modes in the future as well ?

To a degree yes, as the code foundation for choosing different scanning modes is now there. Of course the much bigger task would still be actually implementing the other scanning modes (the 'high precision' scanning mode is a relatively minor change to the default scanning mode.)

Regarding the netflix scanning mode, as suggested above, you could do a feature request at libebur128 directly for optionally disabling the gating, I mean can't hurt me thinks. If they approve that would save us doing this part (modifying the library) ourselves at least.

nofishonfriday added a commit to nofishonfriday/sws that referenced this issue Oct 18, 2018
…ance

see reaper-oss#1046 (comment)
- Revert changing relative gating threshold from -10.0 LU to -20.0 LU
- Add action 'SWS/BR/NF: Toggle use high precision mode for loudness analyzing'
- Harden 'prevent items going offline during analysis' (also prevent during quick reanalyze)
@AironAudio
Copy link
Author

@AironAudio AironAudio commented Oct 25, 2018

@nofishonfriday
Copy link
Collaborator

@nofishonfriday nofishonfriday commented Nov 4, 2018

Did some work on it recently.
I built a static library (.lib) from the Dolby source which SWS can be linked against when building.
The dialogue detection / gating already basically works when doing Loudness scanning.

@cfillion (or anyone):

Another possibility is to make the dolby code available to SWS's existing loudness code via a DLL

So do I have it right that providing the Dialogue Intelligence lib's (32 and 64 bit) and the corresponding include (header) files to the SWS repo should be ok?
Best would probably be to get the approval (or not) from @swstim, because in negative case I wouldn't continue working on this.

@cfillion
Copy link
Collaborator

@cfillion cfillion commented Nov 5, 2018

Adding compiled binaries (like .lib) to a source code repository isn't good practice (I know we already have pre-compiled taglib hosted here, ugh...). Ideally dependencies should be built along or before SWS.

I don't think Dolby's license terms (#1046 (comment)) allows us to copy their headers either... I think it should work like we do with WDL: downloaded separately.

I suggested making it a DLL because I don't know if we can safely build it straight into SWS. The DLL is no better, but at least it wouldn't be part of the SWS DLL itself and could be made as an optional non-open source addon.

@nofishonfriday
Copy link
Collaborator

@nofishonfriday nofishonfriday commented Nov 5, 2018

Thanks.
I got the idea doing a static library when I saw taglib hosted here indeed.
But if you say it isn't good practice, ok.
To have the Dolby source code downloaded separately doesn't seem very feasable to me either as it's not as straigh forward as with WDL, just cloning a repo vs. signing an agreement which maybe not everyone wants to do working on SWS, then probably SWS build cases would have to be made (is the Dolby code present or not) and so on that I think it's too much hassle continuing working on this (what concerns me at least.) Pitty...

nofishonfriday added a commit to nofishonfriday/sws that referenced this issue Nov 7, 2018
…ance

see reaper-oss#1046 (comment)
- Revert changing relative gating threshold from -10.0 LU to -20.0 LU
- Add action 'SWS/BR/NF: Toggle use high precision mode for loudness analyzing'
- Harden 'prevent items going offline during analysis' (also prevent during quick reanalyze)
@nofishonfriday
Copy link
Collaborator

@nofishonfriday nofishonfriday commented Nov 9, 2018

I'm too motivated to get this in SWS to give up just yet. :)
It can now be built along with building SWS directly from the Dolby source (no pre-compiled .lib or .dll) by putting the source in a dedicated SWS directory (not under version control) and I've used #ifdef's so SWS can be built with or without the Dialogue scanner, depending on if the source is present or not.

@swstim / @cfillion
Does this sound like a viable approach?

Sneak peak:
The test file that's included with the Dolby source (containing dialogue and music), scanned with VisLM ('Netflix' preset) and the 'SWS Dialogue loudness scanner".

gif

@AironAudio
Copy link
Author

@AironAudio AironAudio commented Nov 10, 2018

Good grief, you got it going already?! What an accomplishment. I'm still learning C++ over here in the dunce corner.

Nofishonfriday , what test files did you use. I can use the NuGen LM Correct software to verify more test files if you like. I've got plenty of test files.

Since the license in the SWS source code is likely not compatible with the Dolby license, would a separate DLL download be a whole lot more work ? The SWS site, or Github could host it, but the source would need to stay private I will presume.

Maybe some folks could rewrite all parts of it to make it their own and put that out as open source. That could be a good challenge down the road. The library is useful for speech detection. To that regard, providing functions to Reaper scripting users could prove beneficial. Who knows what people could put together with reliable speech detection. It could help to develop an exceptional, almost level-independant noise gate for noisy speech sources for example.

Also, other DSP-savy folks could optimize this in a good way too. The source documentation already mentions spots for optimization.

@nofishonfriday
Copy link
Collaborator

@nofishonfriday nofishonfriday commented Nov 11, 2018

@AironAudio
If you're interested in testing, I can PM you a test version soon(ish), so we can see if it's even worth continuing this project.
First test results look quite ok here, but I have limited ('real life') test material.

@AironAudio
Copy link
Author

@AironAudio AironAudio commented Nov 11, 2018

I've got plenty. I'll gladly give it a run with a few dozen mixes and compare results to the NuGen tool.

For final distribution there seem to be at least two solutions. A binary-only download, which is perfectly legal if there's no code in the binary that requires making the source code available. Personally that would be my least favorite choice, since our community thrives on access to new tools, and that would deny many that access.

The second idea already expressed it to have just the ddi(dolby dialogue intellience) library as a binary-only download, source available on request on a private Github. The SWS extension would check for its existence and offer certain functions if that is available. That's probably a lot more work though.

But we could ask for help from other community members in providing such features for other occasions as well. For example, SWS could provide a frontend to the FFMPEG or Handbrake encoder to provide multi-core video rendering to other format. There too, the commandline executables would need to be available and thus functions be provided to the user.

Then again, maybe those kinds of features(checking & have functions available or work/not-work are already in the extension, and they can be used or copied for use with the ddi library.

So, when you'd like some testing done, I'll run a huge stack of mixes through both the extension and the Nugen tool and compare results. Please keep in mind that we don't have to be identical. Your GIF promises that all is already just fine. We have a +/- 3dB playground in the spec. Actually I'm starting to think this is a better spec for spoken word programs. It works rather well so far.

nofishonfriday added a commit to nofishonfriday/sws that referenced this issue Dec 10, 2018
…ance

see reaper-oss#1046 (comment)
- Add action 'SWS/BR/NF: Toggle use high precision mode for loudness analyzing'
- Harden 'prevent items going offline during analysis' (also prevent during quick reanalyze)
- Internal cleanups (remove silly 'public passthrough' functions)
nofishonfriday added a commit to nofishonfriday/sws that referenced this issue Dec 10, 2018
…ance

see reaper-oss#1046 (comment)
- Add action 'SWS/BR/NF: Toggle use high precision mode for loudness analyzing'
- Harden 'prevent items going offline during analysis' (also prevent during quick reanalyze)
- Internal cleanups (remove silly 'public passthrough' functions)
nofishonfriday added a commit to nofishonfriday/sws that referenced this issue Dec 10, 2018
…ance

see reaper-oss#1046 (comment)
- Add action 'SWS/BR/NF: Toggle use high precision mode for loudness analyzing'
- Harden 'prevent items going offline during analysis' (also prevent during quick reanalyze)
- Internal cleanups (remove silly 'public passthrough' functions)
@maximehyh
Copy link

@maximehyh maximehyh commented Jan 24, 2019

Hi @AironAudio, hi @nofishonfriday,

First, thanks a lot for opening this issue as I was starting to feel lonely looking for a way to implement Dolby's algorithm.

We currently do it using Dolby Media Meter but I am looking for a way of doing the exact same measurement using Python and potentially packaging the whole thing in a docker container.

@nofishonfriday your results shown in the gif look excellent and I was wondering if you could consider PM the backend code (I have some limited knowledge in C++) thats makes the loudness measurement using the speech gating algorithm? I did read Dolby's documentation but that could help me a lot in order to better understand the processing workflow (K filtering, means...) and when/where i need to add the Dolby processing.

I would also be interested in contributing to a broader accessibility of Dolby's algorithm within loudness measurement.

Thanks!

@nofishonfriday
Copy link
Collaborator

@nofishonfriday nofishonfriday commented Jan 24, 2019

@maximehyh

Hi,
for the loudness measurement we use libebur128 library which takes care of the K filtering and all the other Loudness measurement stuff.
Note that because libebur128 uses BS-1770.4 standard which applies gating it must be modified (remove the gating) to work with measuring Dialogue Loudness (see @AironAudio's first comment here).

My current Dialogue Loudness scanning implementation in sws extensions you can find here (NF_DialogueLoudness.h/.cpp), specifically see AnalyzeData() where the actual speech gating using Dialogue Intelligence and following Loudness measuremnt using libebur128 is done (I tried to comment as good as I could :)).
Note it's work in progress currently (but it seems to work quite ok already, we did some more tests meanwhile).

Hope this helps.

@maximehyh
Copy link

@maximehyh maximehyh commented Jan 24, 2019

Excellent! Thanks a lot for the links, I'll start diving into it right away.

One quick question: I would like to try something like AnalyzeData("PATH_TO_MY_FILE") and I am wondering what exactly is a MediaTrack object? I guess it is related to sws?

@nofishonfriday
Copy link
Collaborator

@nofishonfriday nofishonfriday commented Jan 24, 2019

what exactly is a MediaTrack object?

sws is an extension for the REAPER digital audio workstation (as you may know), which has a quite extensive API. A MediaTrack is a pointer to a media track in a REAPER project which can be retrieved using the API function GetTrack().

@maximehyh
Copy link

@maximehyh maximehyh commented Jan 25, 2019

Alright thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants