Video stream orientation the receiver side

[brunosiqueira]

I've been trying to stream videos to wowza in portrait mode. Setting preview orientation doesn't seen to have any effect. Is there any correction available to make it work?

[fyhertz]

Hi,

You would need to rotate the video on the receiver side! There is nothing implemented right now to rotate the video before encoding it.

When using the MediaRecorded API (On android ICS) to get encoded video from the phone, the video is recorded in lanscape. When recording to a file, the API adds some information to indicated to the decoder that the video has to be rotated, when using this API for streaming, this information is lost (It should be be in the moov atom of the MP4 file...)

When using the MediaCodec API with the Buffer to Buffer method, you could try to rotate the NV21 images that you get from the camera but my guess is that you can't do that quickly enough with plain java, maybe a render script (GPU only used on some phone from what I read) or some C (JNI is slow though) could do the trick.

The best hope to achieve this would be to use the MEdiaCodec API with the surface to buffer method, did you check if that was working on your phone?

You will need this to try it:
mSession.getVideoTrack().setStreamingMethod(MediaStream.MODE_MEDIARECORDER_API_2);

In that case, it may be possible to ask the GPU to rotate the texture obtained from the camera, I don't know how though, but it may not be complicated with the right OpenGL calls... If someone could add this it would be awesome.

[Livinlawrence]

Hi i want the recorded video in portrait mode, i did change mSession.getVideoTrack().setStreamingMethod(MediaStream.MODE_MEDIARECORDER_API); and the VideoStream.java by adding the following line mMediaRecorder.setOrientationHint(90);. Now i'm getting an error on mMediaRecorder.setOutputFile(mSender.getFileDescriptor()); All i require is to play this landscape video in portrait mode at the server side.

[brunosiqueira]

I haven't been able to solve this problem yet either...

[Livinlawrence]

Is there anyone here has a solution for this issue..?
I have been with this problem for the last one month. Please do help me to resolve the issue

[iphoneandroidtesting]

Is anyone solved this issue ?
Please help.I am also struck from many days

[brunosiqueira]

Still no solution...

[alexcohn]

I am afraid that Wowza ignores the orientation hints for incoming video, therefore if you cannot control the receiver side, you are stuck with one of the methods described above.

[Livinlawrence]

@alexcohn It is really hard with OpenGL calls and since i'm a beginner to the android i'm not so sure about such stuff. Moreover fyhertz is not sure about that method. Are you able to find a solution with that method.

[vaibhavTesting]

Hi Friends!!

Did you got any solutions for the same? Also want to display live broadcasted video in portrait mode while streaming .

Please suggest and share if you guys having any solution in OpenGL also..

[vaibhavTesting]

Hi Friends,

When I broadcast any live video from android and see the video on other side its resolution not looks good, can you please suggest to improve resolution of video.

Thanks,

[henry12]

any solution?

[cvofjaspreet]

any solution ?

[cvofjaspreet]

In VideoStream class change mOrientation = 90;

[tneginareb]

Explore "libstreaming" library then go to:

1. VideoStream.java class
2. Find createCamera() method, replace "mCamera.setDisplayOrientation(mOrientation);"
   with mCamera.setDisplayOrientation(90);
3. Find updateCamera(), replace "mCamera.setDisplayOrientation(mOrientation);"
   with "mCamera.setDisplayOrientation(90);"

[grooveslap]

Thank you tneglnareb but unfortunately it didn't rotate or change anything when I tried as you guided.

[cvofjaspreet]

Hi Guys,

I have resolved it by rotating the frame before sending it.I have made the camera full screen and with portrait stream.

Here is my lib
https://www.dropbox.com/s/asgko3348kut2bw/net.zip?dl=0

I am using android basic surface view

<android.support.v4.view.ViewPager
android:layout_width="fill_parent"
android:layout_height="fill_parent"
/>

Mail me if you are facing any issue : cvofjaspreet@gmail.com, jaspreet.android@gmail.com
Jaspreet

[grooveslap]

@cvofjaspreet: Thank you!

I got this on Serverside (Wowza server) when trying to use the lib you have linked above. It was OK with the original libstreaming before. Also I see errors (null point & camera is being used after released) in Android Studio Logcat. You have any idea? BTW I have commented out a couple of lines in VideoStream.java (related to ChatActivity)

-16 19:57:03.366 26360-27345/kr.co.mediapeople.rtsptestclient W/System.err﹕ java.lang.NullPointerException: Attempt to invoke virtual method 'void android.media.MediaRecorder.stop()' on a null object reference
07-16 19:57:03.367 26360-27345/kr.co.mediapeople.rtsptestclient W/System.err﹕ at net.majorkernelpanic.streaming.MediaStream.stop(MediaStream.java:293)
07-16 19:57:03.367 26360-27345/kr.co.mediapeople.rtsptestclient W/System.err﹕ at net.majorkernelpanic.streaming.video.VideoStream.stop(VideoStream.java:273)
07-16 19:57:03.367 26360-27345/kr.co.mediapeople.rtsptestclient W/System.err﹕ at net.majorkernelpanic.streaming.video.VideoStream.stopPreview(VideoStream.java:304)
07-16 19:57:03.367 26360-27345/kr.co.mediapeople.rtsptestclient W/System.err﹕ at net.majorkernelpanic.streaming.video.VideoStream.switchCamera(VideoStream.java:137)
07-16 19:57:03.367 26360-27345/kr.co.mediapeople.rtsptestclient W/System.err﹕ at net.majorkernelpanic.streaming.Session$7.run(Session.java:599)
07-16 19:57:03.367 26360-27345/kr.co.mediapeople.rtsptestclient W/System.err﹕ at android.os.Handler.handleCallback(Handler.java:739)
07-16 19:57:03.367 26360-27345/kr.co.mediapeople.rtsptestclient W/System.err﹕ at android.os.Handler.dispatchMessage(Handler.java:95)
07-16 19:57:03.367 26360-27345/kr.co.mediapeople.rtsptestclient W/System.err﹕ at android.os.Looper.loop(Looper.java:135)
07-16 19:57:03.367 26360-27345/kr.co.mediapeople.rtsptestclient W/System.err﹕ at android.os.HandlerThread.run(HandlerThread.java:61)
07-16 19:57:03.446 26360-29203/kr.co.mediapeople.rtsptestclient I/callback﹕ callback
07-16 19:57:03.700 26360-29203/kr.co.mediapeople.rtsptestclient I/callback﹕ callback
07-16 19:57:03.702 26360-29203/kr.co.mediapeople.rtsptestclient W/System.err﹕ java.lang.RuntimeException: Camera is being used after Camera.release() was called
07-16 19:57:03.703 26360-29203/kr.co.mediapeople.rtsptestclient W/System.err﹕ at android.hardware.Camera.native_getParameters(Native Method)
07-16 19:57:03.703 26360-29203/kr.co.mediapeople.rtsptestclient W/System.err﹕ at android.hardware.Camera.getParameters(Camera.java:1888)
07-16 19:57:03.703 26360-29203/kr.co.mediapeople.rtsptestclient W/System.err﹕ at net.majorkernelpanic.streaming.video.VideoStream$1.onPreviewFrame(VideoStream.java:460)
07-16 19:57:03.703 26360-29203/kr.co.mediapeople.rtsptestclient W/System.err﹕ at android.hardware.Camera$EventHandler.handleMessage(Camera.java:1112)
07-16 19:57:03.703 26360-29203/kr.co.mediapeople.rtsptestclient W/System.err﹕ at android.os.Handler.dispatchMessage(Handler.java:102)
07-16 19:57:03.703 26360-29203/kr.co.mediapeople.rtsptestclient W/System.err﹕ at android.os.Looper.loop(Looper.java:135)
07-16 19:57:03.703 26360-29203/kr.co.mediapeople.rtsptestclient W/System.err﹕ at net.majorkernelpanic.streaming.video.VideoStream$2.run(VideoStream.java:564)
07-16 19:57:03.703 26360-29203/kr.co.mediapeople.rtsptestclient W/System.err﹕ at java.lang.Thread.run(Thread.java:818)

[grooveslap]

@cvofjaspreet

Or you could simply let me know where to change in source code in order to make camera rotate.

Thanks in advance.
David

[cvofjaspreet]

Hi,

Here is code to rotate.

In VideoStream.java

Camera.PreviewCallback callback = new Camera.PreviewCallback() {
long now = System.nanoTime()/1000, oldnow = now, i=0;
ByteBuffer[] inputBuffers = mMediaCodec.getInputBuffers();
@OverRide
public void onPreviewFrame(byte[] data1, Camera camera) {
try {
Log.i("callback", "callback");
Camera.Size size = mCamera.getParameters().getPreviewSize();
byte[] data=new byte[data1.length];
rotateNV21(data1,data,size.width,size.height,90);
oldnow = now;
now = System.nanoTime() / 1000;
if (i++ > 3) {
i = 0;
//Log.d(TAG,"Measured: "+1000000L/(now-oldnow)+" fps.");
}
try {
int bufferIndex = mMediaCodec.dequeueInputBuffer(500000);
if (bufferIndex >= 0) {
inputBuffers[bufferIndex].clear();
if (data == null)
Log.e(TAG, "Symptom of the "Callback buffer was to small" problem...");
else convertor.convert(data, inputBuffers[bufferIndex]);
mMediaCodec.queueInputBuffer(bufferIndex, 0, inputBuffers[bufferIndex].position(), now, 0);
} else {
Log.e(TAG, "No buffer available !");
}
} finally {
mCamera.addCallbackBuffer(data);
}
}catch(Exception exception){
exception.printStackTrace();
}
}

    };

public static void rotateNV21(byte[] input, byte[] output, int width, int height, int rotation) {
boolean swap = (rotation == 90 || rotation == 270);
boolean yflip = (rotation == 90 || rotation == 180);
boolean xflip = (rotation == 270 || rotation == 180);
for (int x = 0; x < width; x++) {
for (int y = 0; y < height; y++) {
int xo = x, yo = y;
int w = width, h = height;
int xi = xo, yi = yo;
if (swap) {
xi = w * yo / h;
yi = h * xo / w;
}
if (yflip) {
yi = h - yi - 1;
}
if (xflip) {
xi = w - xi - 1;
}
output[w * yo + xo] = input[w * yi + xi];
int fs = w * h;
int qs = (fs >> 2);
xi = (xi >> 1);
yi = (yi >> 1);
xo = (xo >> 1);
yo = (yo >> 1);
w = (w >> 1);
h = (h >> 1);
// adjust for interleave here
int ui = fs + (w * yi + xi) * 2;
int uo = fs + (w * yo + xo) * 2;
// and here
int vi = ui + 1;
int vo = uo + 1;
output[uo] = input[ui];
output[vo] = input[vi];
}
}
}

In MediaStream.java

/** Starts the stream. */
public synchronized void start() throws IllegalStateException, IOException {

    if (mDestination==null)
        throw new IllegalStateException("No destination ip address set for the stream !");

    if (mRtpPort<=0 || mRtcpPort<=0)
        throw new IllegalStateException("No destination ports set for the stream !");

    mPacketizer.setTimeToLive(mTTL);

    if (mMode != MODE_MEDIARECORDER_API) {
        encodeWithMediaCodec();
    } else {

// encodeWithMediaRecorder();
encodeWithMediaCodec();
}
}

[cvofjaspreet]

Hi,

You must be using old version of lib.
This line is part of lib.

Has

Sent from my iPhone

On 16-Jul-2015, at 5:15 pm, grooveslap notifications@github.com wrote:

@cvofjaspreet

I've got an syntax error when I copy & paste your code into mine:

                else convertor.convert(data, inputBuffers[bufferIndex]);

What's convertor? Is this a variable? There's no declaration in your code.

Thanks.

—
Reply to this email directly or view it on GitHub.

[grooveslap]

Hi Has,

Thank you so much. Indeed it works perfectly!

One thing I am still working on is to prevent mirrored frames when I use front-facing camera. If I do 90 degree rotation with video frames generated from the front-facing camera, on the receiving-end, video frames are upside down. So, wondering if you have any idea or a solution for this.

Anyway, it's been great help already! Thanks in advance.
David

[cvofjaspreet]

Hi David,

I am working on Front camera. I will let you know If I face the issue,but you can modify that method to rotate 180.

@brunosiqueira can you close the issue ?

Jas

[michelrbr]

Hi Guys,

Thank you, @cvofjaspreet your code helped me a lot.
@grooveslap I solved the front camera issue with this code.
...
int rotation = (mCameraId == Camera.CameraInfo.CAMERA_FACING_FRONT) ? 270 : 90;
rotateNV21(data1,data,size.width,size.height,rotation);
...

But I got another issue. When I switch the camera, the stream lose the image, but the audio still works.
Any idea @cvofjaspreet ?

Michel

[cvofjaspreet]

Hi Guys,

@michelrbr I have not worked on front camera till now.

Any buddy set video quality dynamically depending on device camera supported resolutions ? I have set it static
public final static VideoQuality DEFAULT_VIDEO_QUALITY = new VideoQuality(640,480,15,500000);

Jas

[jackthorn]

Hey @cvofjaspreet
Did you manage to find a solution for media recorder?

[ucdevinda123]

@cvofjaspreet you code is working fine but it is not that stable it seems.Specially when you are changing resolutions video does not stream properly to the player(like above screen). Is it possible to get a more robust solution for this. Also is there any solution for front camera issue ??

[jaspreetWegile]

Hi Guys,

I was also having few issues at that time but now I am done with the project.

I can't share the code as its confidential with my client.

My app
https://play.google.com/store/apps/details?id=com.wingtrill

We have not done any advertisements as we are still testing it.

I can help you if you are facing issue and that issue is not there in my app.

My Skype
cvofjaspreet

[jyfree]

@cvofjaspreet I try your way but the picture's color is close purple when you decode

[dvidz]

Hi guys,

The code provided by @cvofjaspreet works, the images is rotating but the with still the same.
For example, using 177 x 144, image must be rotating and become 144x177.

Anyone have a solution to obtain fully rotaed image 177x144 ?

I've been working with libyuv but didn't help a lot.

[sbaar]

@dvidz Do you want to rotate the frame by 90 or 180? Here is the code I enable when the preview orientation is 180 or 270. I don't mind if the picture is mirrored since 95% of users use surface method.

https://gist.github.com/sbaar/0a85727d599fcbd5acd1

[dvidz]

@sbaar I m using This fonction to rotate the bytes but if anyone Knowles how to deal with to inverse width and heigth with This, it would be very usefull.
Also il tryed tout rotate matrix in opengl fonction without any success.

Hey guys have u found the solution ?

[sbaar]

If, for example, the preview is 640x480 in portrait, then setting the rotation to 90 will cause the opengl to receive a 480x640.

[ucdevinda123]

Gave up this Lib

[findxain]

hi @cvofjaspreet u know u are a life savor wish i could do some thing for u. Its ur Day today :) 👍

[GauravPandey123]

Lib streaming library is not working with marshmallow how can we solve it. Actually i am not getting where is issue my code is looking fine but i didn't get what is actually issue.

Please Help.

[ghost]

@dvidz I finally solved the problem, you must set the quality 144x177 and then in @cvofjaspreet solution change the rotateNV21 method in this way: get the camera frame that is still 177x144 convert it to bitmap and then rotate it then again convert it to NV21 and put it in the buffer ;)

[ghost]

@cvofjaspreet thanks in advance for your last solution. Is there any similar solution to manipulate the frame in API2 ? as I investigated we can not use dequeueInputBuffer() in method 2, so how we can do the same thing in method2?