Getting distorted image after converting RawImage to System.Drawing.Bitmap

[mail2mhossain]

I am using SIPSorceryMedia.FFmpeg as a media library and Janus as media Server.
When I am converting the RawImage of Publisher's Camera feed got from event OnVideoSourceRawSampleFaster to System.Drawing.Bitmap image = new Bitmap(rawImage.Width, rawImage.Height, rawImage.Stride, System.Drawing.Imaging.PixelFormat.Format24bppRgb, rawImage.Sample), converted image is OK.

But when I am converting the RawImage of Subscriber's feed got from event OnVideoSinkDecodedSampleFaster to System.Drawing.Bitmap image = new Bitmap(rawImage.Width, rawImage.Height, rawImage.Stride, System.Drawing.Imaging.PixelFormat.Format24bppRgb, rawImage.Sample), converted image is distorted.

Please provide me some clue to solve the issue.

[sipsorcery]

First thing I'd try is a diffent pixel format instead of Format24bppRgb, 32bpp being the obvious one.

If the image looks correct but has a slice curt off then you have the Stride wrong.

[mail2mhossain]

Is it a right way to calculate expected Stride, then convert it to bitmap

int expectedStride = rawImage.Width * 3;
if (rawImage.Stride == expectedStride)
{
Bitmap image = new Bitmap(rawImage.Width, rawImage.Height, rawImage.Stride, PixelFormat.Format32bppRgb, rawImage.Sample);
}

[mail2mhossain]

After implementing above code, now getting black and white frame in both (own video and remote video)

Still remote video is distorted.

[sipsorcery]

Can you paste in some images with the options you applied to them.

[mail2mhossain]

Please find the video in the attachment.

Bitmap image = new Bitmap(rawImage.Width, rawImage.Height, rawImage.Stride, PixelFormat.Format24bppRgb, rawImage.Sample);

Video.zip

[sipsorcery]

Is the remote video meant to be the same as Local video?

In the remote video there seem to be a few frames that are good. Is that the case?

[mail2mhossain]

Yes, remote video is same as local. Yes, few frames are good from remote.

…

On Sun, Jan 21, 2024, 4:56 PM Aaron Clauson ***@***.***> wrote: Is the remote video meant to be the same as Local video? In the remote video there seem to be a few frames that are good. Is that the case? — Reply to this email directly, view it on GitHub <#69 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABUV2TBZVZG32UMGDYWY5XTYPTX65AVCNFSM6AAAAABB6FS7KWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBSGU4TEMZQGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[sipsorcery]

If there is even a single good frame coming through then it's not an encoding issue. More likely there are packets being dropped and key frames are missing.

Try reducing the frame rate and/or resolution and see if you can get a stable video stream. After that you can gradually increase to see where the problem starts.

[mail2mhossain]

I've adjusted the frame rate to 15 using the following code in the InitialiseDecoder method of the FFmpegVideoEncoder class, but I'm still experiencing the same outcomes.

_decoderContext->time_base.den = 15;
_decoderContext->time_base.num = 1;
_decoderContext->framerate.den = 1;
_decoderContext->framerate.num = 15;
_decoderContext->gop_size = 15;

The remote video, which we're receiving through the Janus media server, seems to be having issues. However, the local video is functioning well with the default frame rate of 30.

Publishing video with following code:
var cameraSource = new FFmpegCameraSource(WebCamName);
cameraSource.RestrictFormats(x => x.Codec == _VideoCodec); //_VideoCodec = VideoCodecsEnum.VP8;
cameraSource.OnVideoSourceRawSampleFaster += _videoSource_OnVideoSourceRawSampleFaster;

var videoTrack = new MediaStreamTrack(
cameraSource.GetVideoSourceFormats(),
MediaStreamStatusEnum.SendOnly);

_pub_peerConnection.addTrack(videoTrack);
cameraSource.OnVideoSourceEncodedSample += _pub_peerConnection.SendVideo;

_pub_peerConnection.OnVideoFormatsNegotiated += (formats) =>
{
_negotiatedVideoFormat = formats.First();
cameraSource.SetVideoSourceFormat(_negotiatedVideoFormat);
};

[Gartarr]

I have the same issue, no matter what format I choose, the Rgb24 format produces less distorted photo than the others, but still very much distorted one.

[mail2mhossain]

We're currently setting up a new object called FFmpegVideoEndPoint(). Inside the constructor of FFmpegVideoEndPoint, it is creating another object of class FFmpegVideoEncoder.

In the InitialiseDecoder method of the FFmpegVideoEncoder class, we're observing the following settings:

Bit Rate: 200000
Pixel Format: AV_PIX_FMT_NONE
Frame Rate Numerator: 0
Frame Rate Denominator: 1
Time based Numerator: 1
Time based Denominator: 1
Gop Size: 12

After running the code ffmpeg.avcodec_open2(_decoderContext, codec, null).ThrowExceptionIfError();, our settings change to:

Bit Rate: 200000
Pixel Format: AV_PIX_FMT_YUV420P
Frame Rate Numerator: 0
Frame Rate Denominator: 1
Time based Numerator: 1
Time based Denominator: 1
Gop Size: 12

This leads me to a few questions:

1. We notice that the Pixel Format is consistently AV_PIX_FMT_YUV420P. Could this be causing image distortion, especially since we are using System.Drawing.Imaging.PixelFormat.Format24bppRgb?
2. The Bit Rate always remains at 200000. I'm wondering if this could also be a factor in the image distortion.
3. The Frame Rate is always zero. Do you think this could have any impact?

I'd appreciate any insights or suggestions you might have on these points. Thanks for your help!

[mail2mhossain]

We are currently receiving the bit rate dynamically from another client.
However, after executing the code ffmpeg.avcodec_open2(_decoderContext, codec, null).ThrowExceptionIfError();, we find that the bit rate value is set to 200,000.
We are looking to capture the bit rate dynamically. Does the SIPSorceryMedia.FFmpeg library support this functionality?

[ha-ves]

@mail2mhossain
We are currently receiving the bit rate dynamically from another client. However, after executing the code ffmpeg.avcodec_open2(_decoderContext, codec, null).ThrowExceptionIfError();, we find that the bit rate value is set to 200,000. We are looking to capture the bit rate dynamically. Does the SIPSorceryMedia.FFmpeg library support this functionality?

Can you try using PR #62 ? It allows dynamic bitrate by reloading the encoder. (FFmpeg Encode/VP8 Variable Bitrate explanation)