Gaze tracking is not accurate

[begaiym-k]

Hi! First of all thanks for publishing your work, it is really helpful! I have been working on my graduation project which is very similar to yours and I wanted to run your program to get some idea on how it works.

I was able to calibrate and get the yaml file, then I ran the main,py and manually set entered the screen sizes. However, when I run it the screen appears to be way smaller then the actual screen (playing with the numbers didn't help), the laser doesn't appear , and it shows the red line mostly in the center unless I move my head than the line starts moving towards the side my head moved.
Can it be because of the calibration? Or what might cause it? How did it work for you, can you share with more details please?

Thank you!

[pperle]

Hey @begies-projects,
thank you for your issue.

So, if I understand it right, you successfully calibrated your camera and now want to try out the demo.

when I run it the screen appears to be way smaller then the actual screen

Do you mean in the 3D visualization of the environment? Is your face plotted in the 3D visualization? The ray intersecting with the screen in the 3D visualization should directly correspond with the "lasterpointer".
If you are using Linux, you can try to get the parameters from the system

gaze-tracking-pipeline/utils.py

Line 9 in be33d7e

def get_monitor_dimensions() -> Union[Tuple[Tuple[int, int], Tuple[int, int]], Tuple[None, None]]:

, but this only works when you are using a single monitor. Where is your camera located in relation to the screen the code expects the camera to be in the center on top of the screen.

it shows the red line mostly in the center unless I move my head

That sounds like the model always predicts the same values. Maybe log the predications here and see if there is something wrong. You can also visualize the preprocessed images with the --visualize_preprocessing flag to validate that the input to the model is correct.

Can it be because of the calibration?

I could be, but even without calibration, the model should predict the rough direction. If you collect calibration data (images of the face + actual point on the screen) you could also visualize this data to make sure that the setup is correct. Have a look at this repository, where I have already written the necessary code.

[begaiym-k]

Hello @pperle
Thanks a lot for your response and suggestions, I really appreciate it. I have tried different things with this algorithm and the model and here what I have got.

I ran the --visualize_processing and it seems correct since I can see my face, left and right eyes. I logged the predictions, and I can see something an array, I don't know if this is what was supposed to give?

[-0.10606598 0.08103976] [-0.09040515 0.06267491] [-0.11975915 0.06924455] [-0.23429006 0.02587351] [-0.21347389 0.03769534] [-0.22900468 0.01540647] [-0.3816366 0.01652036] [-0.37407577 -0.00279256] [-0.37453973 -0.00583977]

On the main screen with a red laser pointer I can only see a part of the screen, so I ran get_monitor_dimensions() but it gives such small values that the screen shows up even smaller, when I enter the dimensions manually it still is not a full size. I have attached the screenshot from my computer.

Another thing is that as you mentioned the model mostly predicts only the central values, so that the red laser is in the center almost all the time. I looked at the gaze calibration data collection and even implemented it but there it cannot complete the data collection. I looked at the black screen with E letter and pressed the arrow keys for almost 10 mins and it still didn't finish, is that normal?

Are there any suggestions on how I can improve the model output/predictions so that I can get at least the relative points on the scree?

[hshahid]

@begaiym-k did you figure this out?

[emrecolak55]

@begaiym-k Could you solve this issue?

[begaiym-k]

@emrecolak55, @hshahid no, unfortunately I couldn't make the software fully functional.