Received green layer overlaps the video frame when running Android device.

[thieplehuu]

When i running encode and decode in Android device, I received green layer overlaps the video frame. Such as image bellow.

When I shake the device, the image clarity is shown.

In my encode function:
I get NV21 data from android camera after that i convert it to yuv420 and encode it by openh264 EncodeFrame function.

In my decode function:
I use openh264 DecodeFrame function after that i convert yuv420 output data to RGBA format.

I tried it in IOS else encountered a problem so on but color is gray color instead green color. And I tried it in Windows OS else works well.
I don't know where my wrong. Please help me, thank so much.

This is my parametes settings for encode function:

ISVCEncoder* pSVCEncoder; // OpenH264 encoder instance
tUint32 uWidth = 640;
tUint32 uHeight = 480;
UINT uBitrate = 1000000;
UINT uFramerate = 15.0f;
SSourcePicture SrcPic;
UCHAR pcEncInputFrameBuffer[(uWidth * uHeight * 3)/2]; // For Android NV21 to YUV420 conversion
WelsCreateSVCEncoder(&pSVCEncoder);
SEncParamExt encodingParam;
memset(&encodingParam, 0, sizeof(SEncParamExt));
pSVCEncoder->GetDefaultParams(&encodingParam);
encodingParam.iUsageType = CAMERA_VIDEO_REAL_TIME;
encodingParam.iInputCsp = videoFormatI420;
encodingParam.iPicWidth = uWidth;
encodingParam.iPicHeight = uHeight;
encodingParam.iTargetBitrate = uBitrate;
encodingParam.fMaxFrameRate = uFramerate;
encodingParam.iRCMode = RC_OFF_MODE;
encodingParam.iTemporalLayerNum = 1;
encodingParam.iSpatialLayerNum = 1;
int i;
for (i = 0; i < encodingParam.iSpatialLayerNum; i++) {
encodingParam.sSpatialLayers[i].iVideoWidth = uWidth;
encodingParam.sSpatialLayers[i].iVideoHeight = uHeight;
encodingParam.sSpatialLayers[i].fFrameRate = uFramerate;
encodingParam.sSpatialLayers[i].iSpatialBitrate = uBitrate ;
encodingParam.sSpatialLayers[i].uiProfileIdc = PRO_BASELINE; // PRO_SCALABLE_HIGH;
encodingParam.sSpatialLayers[i].uiLevelIdc = LEVEL_3_1;
encodingParam.sSpatialLayers[i].sSliceCfg.uiSliceMode = SM_DYN_SLICE;
encodingParam.sSpatialLayers[i].sSliceCfg.sSliceArgument.uiSliceSizeConstraint = 1300;
}
encodingParam.uiIntraPeriod = uFramerate; /* I-Frame interval in frames */
encodingParam.bEnableSpsPpsIdAddition = 1;
encodingParam.bPrefixNalAddingCtrl = 0;
encodingParam.iPaddingFlag = 0;
encodingParam.uiMaxNalSize = 1300;
encodingParam.bEnableLongTermReference = 1;
encodingParam.iLtrMarkPeriod = uFramerate;
iRet = pSVCEncoder->InitializeExt(&encodingParam);
lLumaSize = (uWidth) * (uHeight);
lChroSize = (lLumaSize >> 2);
memset(&SrcPic, 0, sizeof(SSourcePicture));
SrcPic.iColorFormat = videoFormatI420;
SrcPic.iPicWidth = uWidth;
SrcPic.iPicHeight = uHeight;
SrcPic.iStride[0] = uWidth;
SrcPic.iStride[1] = uWidth >> 1;
SrcPic.iStride[2] = uWidth >> 1;
SrcPic.iStride[3] = 0;
SrcPic.uiTimeStamp = uwTimestamp;
SrcPic.pData[0] = inputBuffer;
SrcPic.pData[1] = SrcPic.pData[0] + lLumaSize;
SrcPic.pData[2] = SrcPic.pData[1] + lChroSize;
SrcPic.pData[3] = NULL;
memset(&info, 0, sizeof(SFrameBSInfo));

[luuanhtu]

I also get into this problem. Please help us to confirm the issue soon.

1. For windows application, just wrapper the encoder and decoder as Transform Filters (DSF), then test filter graph as video source → H.264 Encoder Filter → H.264 Decoder Filter → video render, I can see the correct shape and color. The FPS is high enough (15-30fps).
2. Use the same parameter on Android, the result decoded image display correct shape but seem covered by green on background. When quickly shake device, the green seems cannot be drawn fast enough, so that we can see correct color.
3. On iOS the color is not green but grey.

[huili2]

For decoder, do you use DecodeFrame or DecodeFrame2? I suggest use the second one. The console file has an example of that.
Besides, from the picture, there seems to be packet loss happens. Could you please check if that's true?
btw: suggest use the latest version.

[luuanhtu]

Thank you for your answer.

1. I use DecodeFrame2 actually. By the way, for your information, we also are evaluating another H.264 codec and the results are:

• Our H.264 encoder - OpenH264 Decoder: OK
• Open H.264 encoder - Our H264 Decoder: Not good (NG). The decoder cannot completely decode an frame. The error does not happen though.
• Open H.264 encoder - Open H.264 decoder: this issue
• Our encoder - Our decoder: OK

2. We are evaluating on loop-back interface so that the packet loss due to network may not happen.
   However, as your suggestion I may check the encoder parameters related to bitrate, bandwidh, etc whether or not the packet may be skipped at the encoders for bandwidth control of encoder.

3. The source code we are using is relatively new, about 1 week ago.

I will let you know the additional investigation result and hope to get your active support for this issue.

[luuanhtu]

Hi Hulii2,

1. I did checked again the usage of DecodeFrame2 and found no mistake. I am suspecting that the problem is at encoder. Because as you see there is no problem when using our H.264 Encoder and Open H.264 Decoder.

2. For your additional info, I did try with many values of parameters and found that the more frequently you send I frame (uiIntraPeriod param of SEncParamExt), the more the green appears (the picture is not seen clearly due to too much green).
   When I did not set uiIntraPeriod or big value (about 300), the green does exist, but we can recognize sharp somehow in the non-moving state.
   When we move much, the picture is correct both in shape and color.

3. About packet loss, I add counter to encoder and decoder, the difference of them is only 2 frame when I stop.
   By the way I am send packet on the loop back interface. The packet processing flow is
   Android Camera (NV21) → NV21ToYuv420 → Open H.264 Encoder (NAL constrain size mode) →　RTP packetization
   ↓
   Local Network
   ↓
   RTP depacketization → Open H.264 Decoder → Yuv420ToRGBA → Render on Display

Could you mind to confirm the issue and give us hint how to correct it. Your suggestion is highly appreciated.

Regards,
Tu

[huili2]

From what I can consider, the green or grey issue is caused by packet loss, especially for IDR picture, and decoder will insert green or grey (old/new version) for future reference.
I suggest you check the value of DecodeFrame2. if the return value is not 0, some data are lost.
If you have no feedback such as IDR request, encoder would never know packet lost, and keeps on sending P frame.
Setting IDR period is one solution. But as you mentioned, it would not make things better, because given packet loss rate, more IDR causes more loss.

So I would suggest you check if IDR picture has some data lost.
And use latest version.

[luuanhtu]

Your comment is actually very exactly. My bad.
By observing the result of experiment, I supposed that the PPS and SPS are sent in a very special way with Open H.264, that is NALs consisting PPS and SPS are sent without encoded picture information. Then in those NALs, I only save PPS and SPS and send together encoded picture data in next frame.
Now I still do not know why I can discover such a strange and wrong thing!!!

After your comment, I look again the IDR frame, and saw my mistake. The encoded picture data is sent together PPS and SPS in IDR frame, so just send it as whole frame is correct.
Now the Open H.264 encoder and Open H.264 decoder are working well together.
I highly appropriate your analysis. Thank you very much.

[danhuynh0803]

@luuanhtu Hi, I'm seeing a similar issue on my end. Can you clarify what you did to get the green flash to resolve. Can you please include a code sample on how you saved off the PPS and SPS together with the IDR frame?