JeelizHelper tweakMoveYRotateY behaves differently on desktop and mobile

[ThorstenBux]

Describe the bug
I just found that the setting JeelizHelper.tweakMoveYRotateY behaves completely different between mobile (iPhone 6s, Safari) and desktop (macOS Chrome)

To Reproduce
Steps to reproduce the behavior:

1. Go to one of your examples with JeelizHelper
2. Open it on mobile and desktop
3. Wait for detection and nod your head up and down
4. You should notice that the augmented object moves differently between desktop and mobile

Expected behavior
Movement of the object should be the same

Desktop (please complete the following information):
iPhone 6s, Safari

Smartphone (please complete the following information):
macOS
Chrome

Additional context
I had set the value to 2 for desktop and 0.3 for mobile

[xavierjs]

Hi,

This value should be kept at 0.5, otherwise it will produce aberration.
To tune the position of the 3D model you have to play on the position of the model in 3D compared to the container.

In the glasses VTO demo https://jeeliz.com/demos/faceFilter/demos/threejs/VTO/
provided in this repository the glasses are accurately positionned without playing with this parameter.

In your case I think the problem comes from the camera FoV (Field of view) set with threeCamera.fov.
The fov given in the helper and specified in THREE.PerspectiveCamera is the vertical FoV.
In the helper it is the same FoV in both cases. But for a desktop it should be around 40° and for a mobile in portrait mode it should be around 90°.

This issue is related: #56
So you have to make guess about the FoV depending if the device is a desktop, a mobile, in portrait or landscape mode.
There is no web API to get camera calibration parameters.

Best,
Xavier

[ThorstenBux]

Hi @xavierjs thanks for your support and clarification. I know some things about the camera calibration challenges from working on artoolkitX.js I do understand the FoV setting and will adjust it to work better. However, the VTO example you linked to it is set to 40° which works on desktop and mobile very good. So I'm wondering why you are proposing to set it to 90° for mobile?

[xavierjs]

Hi,

The FOV provided for the THREE.js camera and used to compute the position is the vertical FOV.
if you use a 16:9 aspect ratio image then the relation between horizontal fov and vertical fov is 16:9
(if we crop the video it will be a little more complicated).

But anyway, the fov is always the vertical angle of the camera.
A laptop camera is used in landscape mode. If you have an horizontal fov of 70° and you request a 16:9 video, you have a vertical fov of 70/(16/9) = 39°

A mobile in portrait mode have the camera used in portrait mode, so the same camera will provide a vertical fov of 70°.

Then the fov of mobile camera is a bit larger than laptop camera because they are made to be used from a shorter distance, and they should be able to provide landscape pictures. That's why I advise a 90° value.

[ThorstenBux]

Hi @xavierjs thanks for your response:

I've calculated it through. Given the FoV of the front camera released here:
https://developer.apple.com/library/archive/documentation/DeviceInformation/Reference/iOSDeviceCompatibility/Cameras/Cameras.html)
which is horizontal. I'm assuming between 54 - 56 horizontal FOV for front camera on iPhones.

Given a full screen resolution of 375w 553h in Safari and using this resource https://themetalmuncher.github.io/fov-calc/

I end up with a vertical FOV of 74.

I've placed the origin of my objects in the center between the eyes which if I understand correctly should make them appear on the tip of the nose. Because that is what your ML exposes as world origin. Then I move the object up a bit 0.4 - 0/5 to be placed over the eyes.
That works basically quite well. As long as I keep my head still.

Now the tricky part is the rotation I take it that pivotOffsetYZ moves the model origin so that I can set a different pivot point from the actual model origin.

However, no matter what values I'm using the model always appears to be misplaced when rotating the head as if the pivot point is set incorrectly.
Anything you can advise in regards to that?

Thanks

[xavierjs]

Hi,

Indeed the 74° for the FOV seems very plausible.
You can change the position of the pivotOffsetYZ, but it won't be equivalent of changing the position of the object. the offset of the pivot will influence the way the model is moving when you rotate the head.

If you want the best positionning as possible you should just start from the VTO demo (https://github.com/jeeliz/jeelizFaceFilter/tree/master/demos/threejs/VTO).

The glasses are given as OBJ model: https://github.com/jeeliz/jeelizFaceFilter/tree/master/demos/threejs/VTO/models3D

You can import them to help positionning and scaling your own model, then you just replace the glasses by your own model in the demo.

Although the neural network was trained using the center of the 2 eyes as origin, it does not expose its coordinates since they should always be at a fixed place in 2D in the detection window.

[ThorstenBux]

Hi @xavierjs thanks for your reply.
Yes, I expect changes to the pivot only to affect the way the model rotates and not the way the model is positioned.
I've worked with the VTO example first and it is looking good. However, what I can see is that your models have their origin at a different place than mine and I think the simply positioning mine at the same place as yours won't work because of that. Mine would rotate at a different position due to their origin being at another location. That is what I try to compensate with moving the pivotOffsetYZ without luck so far.

[xavierjs]

Yeah, you can either change the positionning of your model according to the 3D glasses and reexport it using blender, or replace the model by a composite object THREE.Object3D having a child with a position offset.

[ThorstenBux]

So you are saying that changing pivotOffsetYZ is not the same as changing the origin of a model?

[xavierjs]

yes it is not the same thing.

[ThorstenBux]

Alright, I think I'm getting closer to my issue:
When I change VTO example to fullscreen mode:

    JeelizResizer.size_canvas({
        canvasId: 'jeeFaceFilterCanvas',
        isFullScreen: true,
        callback: function(isError, bestVideoSettings){
            init_faceFilter(bestVideoSettings);
        }
    })

I get the same effect that I fixed by setting tweakMoveYRotateY.

The result I'm seeing is that the glasses are placed correctly if I look straight at the screen and if I rotate my head around the y-axis.

However, if I rotate the head around the z-axis (nodding) the glasses appear to be too far in the front. (Attached a picture to illustrate the axis I'm talking about)

Any idea why that might be?

[xavierjs]

Have you enabled followZRot option like in the VTO glasses demo? It should fix problem with rotation around Z axis (depth).
( https://github.com/jeeliz/jeelizFaceFilter/blob/master/demos/threejs/VTO/demo.js )
Are you sure you are talking about the Z-axis? Because nodding matches with a rotation around the X axis in your picture.

[ThorstenBux]

Sorry, rotation around x-axis (it is Friday afternoon my mind is already in the weekend I guess).

Yes, followZRot is enabled. Basically I've cloned the VTO example from GitHub and changed the line as shown above here: https://github.com/jeeliz/jeelizFaceFilter/blob/master/demos/threejs/VTO/demo.js#L47

[xavierjs]

In this drawing glasses are drawn from a side view.
I draw the effect of a rotation around X for 3 pivots points differing from their Y coordinate only using different colors.

Currently the Z position of the pivot point is correct since the glasses are well positionned if you rotate the head around Y (left <-> right). The X position of the pivot point should be 0 (because the head is symetric). The problem is with Y.

• If the pivot point is too low (small Y), the glasses will have the impression to go down when rotating the head around X (looking up) -> drawed in green.
• If the pivot point is too hight (large Y), the glasses will go up when rotating the head around X (looking down) -> drawed in red

Currently I think your pivot point is not well positionned along Y. You can compensate this by moving the model, but it won't be the same as soon as the face rotates around X.

[ThorstenBux]

Yes, exactly. And the same happens with your VTO example when fullscreen is enabled. Could you reproduce?
The question is how to fix that.

[xavierjs]

So I think that you should lower the first value of _settings.pivotOffsetYZ in the THREE.js helper: https://github.com/jeeliz/jeelizFaceFilter/blob/master/helpers/JeelizThreejsHelper.js

[ThorstenBux]

If I do change that to

  const _settings = {
    rotationOffsetX: 0, //negative -> look upper. in radians
    pivotOffsetYZ: [0.0,0.2], //[0.2,0.2], //XYZ of the distance between the center of the cube and the pivot. enable _settings.isDebugPivotPoint to set this value
    
    detectionThreshold: 0.75, //sensibility, between 0 and 1. Less -> more sensitive
    detectionHysteresis: 0.05,

    tweakMoveYRotateY: 0.5, //tweak value: move detection window along Y axis when rotate the face
    
    isDebugPivotPoint: true //display a small cube for the pivot point
  };

that fixes it for rotating upwards. (Looking to the sky)
But looking down to the ground the glasses look like too far away from my face.

Still talking about the VTO example.

[xavierjs]

Hi,

I can reproduce the bug. There is definitely something wrong in JeelizFaceFilterHelper. I will figure out what happens. I have my idea about what is going wrong...

[xavierjs]

I think I have found the bug :) .
I did a change in the THREE.js helper ( https://github.com/jeeliz/jeelizFaceFilter/blob/master/helpers/JeelizThreejsHelper.js )

I did not updated the live demos yet. Can you tests with the new helper?

Thank you very much!

[ThorstenBux]

Yes, the new helper makes it much better. I've still got some offset in my prod environment but the VTO example in fullscreen works much better now.

[ThorstenBux]

Many thanks for the quick responses and instant fix @xavierjs

[xavierjs]

Hi @ThorstenBux

I took some days off.
Thank you very much, you helped me to solve a major bug which was here since a while.
If you discover some improvements, or want to share your demo, you are welcome (I can add a link from the readme/third party demo).

Best,
Xavier