Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Extract individual objects? #19

Closed
sclsj opened this issue Oct 14, 2022 · 32 comments
Closed

Extract individual objects? #19

sclsj opened this issue Oct 14, 2022 · 32 comments

Comments

@sclsj
Copy link

sclsj commented Oct 14, 2022

From my reading of other issues it seems like 360 Reality audio downloaded from Tidal is object-based, and during decode it is mapped to multichannel output. Is it possible to extract the individual objects? I really like the track and want to figure out what each instrument does in this as well as recreate parts on piano based on listening, and having objects (I assume one or a group of instruments?) isolated is very helpful for that.

For an example audio, please use this track. If this is not possible, if convenient please let me know of anything you can think of that might help me with this process. Thank you so much for your help!
@SakethSathuvalli
Copy link
Member

Hi,

Please refer to our https://github.com/ittiam-systems/libmpegh/blob/main/docs/IA-MPEG-H-3D-Audio-Dec-GSG.doc and
https://github.com/ittiam-systems/libmpegh/blob/main/docs/IA-MPEG-H-3D-Audio-Dec-API.doc if any information is available for your use case.

Thanks!

@sclsj
Copy link
Author

sclsj commented Oct 14, 2022

I'm sorry, I'm not very literate in this topic. I tried to read the guide but can't really understand it and so I can't find revevant information. Am I correct in understanding that this is not something I can do with the command-line program but something that might be possible by using the library?

@SakethSathuvalli
Copy link
Member

Hi @sclsj,

Please refer to the entries in tables 3-1 and 3-2 of the API document. These tables have details about the buffers that carry the object audio information when external rendering is enabled. The interfaces are provided as per the specification for MPEG-H and expects the user of this data to have prior knowledge of programming and digital audio formats(PCM data of multichannel).

Thanks!

@sclsj
Copy link
Author

sclsj commented Nov 27, 2022

Unfortunately I do not know any programming language. Is there a demo or sample code that allows extraction of object data (i.e. MPEG-H 3d mp4 --> adm bwf) anywhere?

@SakethSathuvalli
Copy link
Member

Hi @sclsj,

We have recently added a wiki page on external rendering interfaces. Can You please check if it helps you with the issue you are facing?

Thanks!

@sclsj
Copy link
Author

sclsj commented Jan 31, 2023

Hi @SakethSathuvalli

It definitely helped. However, it seems like I'm getting a lot of popping noises, perhaps due to some problems with the limiter, for example limiter not applied and resulting in heavy clipping. How can I set a lower global gain (for example, multiple every gain for object by 0.5x so it won't clip)? Or maybe disable DRC for objects completely? I can't find an option for either.

@sclsj
Copy link
Author

sclsj commented Jan 31, 2023

Below is an example. See the first 2 channels that are heavily clipped/limited.

截屏2023-01-31 11 21 26

@sclsj
Copy link
Author

sclsj commented Jan 31, 2023 •
edited
Loading

I think these weird samples are why there are constant popping sounds. It seems like the clipped samples become negative? I'm not sure what's happening but it's definitely because the signal clipped.

截屏2023-01-31 11 21 22

@SakethSathuvalli
Copy link
Member

I think these weird samples are why there are constant popping sounds. It seems like the clipped samples become negative? I'm not sure what's happening but it's definitely because the signal clipped.

截屏2023-01-31 11 21 22

@sclsj, Can You please let know if the waveform shown above is of the audio data file generated using -ext_ren:1 flag ?

@SakethSathuvalli
Copy link
Member

Hi @sclsj,

Try once with our latest commit - 3b23f65

Thanks!

@sclsj
Copy link
Author

sclsj commented Feb 1, 2023 via email

@sclsj
Copy link
Author

sclsj commented Feb 5, 2023 •
edited
Loading

Hi @sclsj,

Try once with our latest commit - 3b23f65

Thanks!

It definitely worked.

However, the objects are still peak limited. How can I set a lower gain for objects?

For the channel output, I messed around with files containing "gain" and lowered the gain (therefore disabling DRC) by modifying all gain in impeghd_peak_limiter.c to 0.1. What would be the corresponding edit that would set a lower object gains (perhaps times a factor of 0.1)?

@SakethSathuvalli
Copy link
Member

Hi @sclsj,

The DRC effect will not be applied on the external renderer interfaces PCM output. It appears that the original content itself is peak limited in nature.

Thanks!

@SakethSathuvalli
Copy link
Member

Hi @sclsj,

We would like to close this issue if You can confirm that you are able to extract individual objects with the -ext_ren: switch.

Regarding the gain, the decoder does not have a provision to apply desired gain on decoded objects.

Thanks!

@SakethSathuvalli SakethSathuvalli mentioned this issue Feb 8, 2023
Closed
@sclsj sclsj closed this as completed Feb 9, 2023
@sclsj
Copy link
Author

sclsj commented Feb 9, 2023

Thank you for all your help! Yes, I was able to use the ext_ren switch. Just want to confirm again that no processing is done to the decoded objects for extren, and all the drc metadata and such are included in the bitstream file?

@sclsj sclsj mentioned this issue Feb 9, 2023
Closed
@sclsj
Copy link
Author

sclsj commented Feb 18, 2023 •
edited
Loading

Hi @sclsj,

Try once with our latest commit - 3b23f65

Thanks!

I'm still curious why that happened. From my experience you can exceed 0 dB in 24 bits? E.g. if I record in QuickTime the peak usually > 0dB and I have to apply like a -10dB gain. Even if 16 bit (e.g. convert from 24 bit to 16 bit in-place in Audition on file with >0dB peak), shouldn't be audio just be clipped? I thought that's the normal behavior?

And also why would that happen in the first place? Shouldn't the encoded objects be peak normalized/limited in the first place (by the encoder)?

And could you explain how did you fix this (e.g. lower the gain?)?

@sclsj
Copy link
Author

sclsj commented Feb 18, 2023

Also, it seems like the new version of libmpegh fails to decode 360 RA files from Deezer.

ia_mpeghd_testbench -ext_ren:1 -ifile:/Users/jin/Music/deemix\ Music/YOASOBI\ -\ THE\ BOOK\ 2/07\ -\ もしも命が描けたら.mp4 -ofile:mingdeemix.wav
-ifile:/Users/jin/Music/deemix Music/YOASOBI - THE BOOK 2/07 - もしも命が描けたら.mp4 -ofile:mingdeemix.wav 
                       ITTIAM SYSTEMS PVT LTD, BANGALORE
                             http:\\www.ittiam.com
                      IA_MPEG_H_3D_AUD_DEC_ARM $Rev: 1.2 $
                                        

fatal error: Ittiam mpegh_dec  core coder module :Initialization: : Initialization fatal error

@SakethSathuvalli
Copy link
Member

Also, it seems like the new version of libmpegh fails to decode 360 RA files from Deezer.

ia_mpeghd_testbench -ext_ren:1 -ifile:/Users/jin/Music/deemix\ Music/YOASOBI\ -\ THE\ BOOK\ 2/07\ -\ もしも命が描けたら.mp4 -ofile:mingdeemix.wav
-ifile:/Users/jin/Music/deemix Music/YOASOBI - THE BOOK 2/07 - もしも命が描けたら.mp4 -ofile:mingdeemix.wav 
                       ITTIAM SYSTEMS PVT LTD, BANGALORE
                             http:\\www.ittiam.com
                      IA_MPEG_H_3D_AUD_DEC_ARM $Rev: 1.2 $
                                        

fatal error: Ittiam mpegh_dec  core coder module :Initialization: : Initialization fatal error

The failure could be because the stream configurations may not be supported by our decoder. Example the stream could belong to a higher profile level.

@SakethSathuvalli
Copy link
Member

Hi @sclsj,
Try once with our latest commit - 3b23f65
Thanks!

I'm still curious why that happened. From my experience you can exceed 0 dB in 24 bits? E.g. if I record in QuickTime the peak usually > 0dB and I have to apply like a -10dB gain. Even if 16 bit (e.g. convert from 24 bit to 16 bit in-place in Audition on file with >0dB peak), shouldn't be audio just be clipped? I thought that's the normal behavior?

And also why would that happen in the first place? Shouldn't the encoded objects be peak normalized/limited in the first place (by the encoder)?

And could you explain how did you fix this (e.g. lower the gain?)?

The fix is related to limiting the upper and lower limits of a 24-bit data. These are referred to as saturation fixes.

@sclsj
Copy link
Author

sclsj commented Feb 22, 2023

Hi @sclsj,
Try once with our latest commit - 3b23f65
Thanks!

I'm still curious why that happened. From my experience you can exceed 0 dB in 24 bits? E.g. if I record in QuickTime the peak usually > 0dB and I have to apply like a -10dB gain. Even if 16 bit (e.g. convert from 24 bit to 16 bit in-place in Audition on file with >0dB peak), shouldn't be audio just be clipped? I thought that's the normal behavior?
And also why would that happen in the first place? Shouldn't the encoded objects be peak normalized/limited in the first place (by the encoder)?
And could you explain how did you fix this (e.g. lower the gain?)?

The fix is related to limiting the upper and lower limits of a 24-bit data. These are referred to as saturation fixes.

Can you tell me a bit more? I failed to get more information with basic search. Did you just clip the samples?

@SakethSathuvalli
Copy link
Member

Hi @sclsj,
Try once with our latest commit - 3b23f65
Thanks!

I'm still curious why that happened. From my experience you can exceed 0 dB in 24 bits? E.g. if I record in QuickTime the peak usually > 0dB and I have to apply like a -10dB gain. Even if 16 bit (e.g. convert from 24 bit to 16 bit in-place in Audition on file with >0dB peak), shouldn't be audio just be clipped? I thought that's the normal behavior?
And also why would that happen in the first place? Shouldn't the encoded objects be peak normalized/limited in the first place (by the encoder)?
And could you explain how did you fix this (e.g. lower the gain?)?

The fix is related to limiting the upper and lower limits of a 24-bit data. These are referred to as saturation fixes.

Can you tell me a bit more? I failed to get more information with basic search. Did you just clip the samples?

Yes, the samples will be clipped to maximum / minimum value if they are found to exceed.

@sclsj
Copy link
Author

sclsj commented Feb 23, 2023 •
edited
Loading

Got it. What should I do if I don't want it to clip? Is it possible to reduce the gain or to output as 32-bit float, which wouldn't have this problem?

@SakethSathuvalli
Copy link
Member

For the individual objects it is not possible to apply gain.

@SakethSathuvalli
Copy link
Member

Even when float output is given the values exceeding [-1.0, 1.0] will be clipped. However, support for float output is currently not available

@sclsj
Copy link
Author

sclsj commented Feb 23, 2023

But is it possible to solve this one way or another? Clipping is still bad, although not nearly as bad as inverted samples.

@sclsj
Copy link
Author

sclsj commented Feb 23, 2023

Even when float output is given the values exceeding [-1.0, 1.0] will be clipped. However, support for float output is currently not available

I'm asking about float since QuickTime Player on macOS records in 24-bit float and every time I record most of the samples are > 0dB. I just do a -12dB volume reduction and it's ready to be used.

@SakethSathuvalli
Copy link
Member

A not so clean way is to tap the output before clipping. This has to be done in the code. Please check the fix given for saturation overflow. It can help you with the buffer containing audio data that is not clipped.

Thanks!

@sclsj
Copy link
Author

sclsj commented Feb 23, 2023

Emm... I just replaced that with pcm_sample = pcm_sample * 0.5f and it seemed to work... Other than increased noise floor, what problems might that cause?

@SakethSathuvalli
Copy link
Member

When You try to apply gain on the output again the clipping behavior will be seen again when the data is stored as 16/24 bit PCM. Other than this I don't see any other implications.

Thanks!

@sclsj
Copy link
Author

sclsj commented Feb 23, 2023 •
edited
Loading

I don't think I understood you. Could you please explain more? Are you saying that if this 0.5 gain is not low enough for certain files, the ext_ren pcm will still be clipped?

Also, why would the encoded objects be greater than 0dB? Shouldn't the encoder clip any values exceeding [-1.0, 1.0]? I don't understand why, if the encoded objects can be greater than 1.0, the decoded output can't be with a significantly higher bitrate. I'm sorry if this question seems silly.

Thank you for all your help!

@SakethSathuvalli
Copy link
Member

All I am saying is when You apply a gain of 2.0 or more on this output(pcm_sample = pcm_sample * 0.5), the samples will get clipped.

The sample values exceed the range [-1.0, 1.0] by a very marginal value(< 1e-2). This is due to the implementation and is expected. Also this can happen even if the encoder takes care of clipping.

Thanks!

@sclsj

This comment was marked as off-topic.

Sign in to view
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@sclsj @SakethSathuvalli