Handpose with FlipVideo Not Working ?

[nikkisingh111333]

hey i m creating a hand gestusre controlled UI for web... everything is fine created my Own custum Cursor using document.elementFromPoints() to get actual position of element but problem is ....want to flip video element before feeding it to Ml5.handPose() its not working
i have also tried options object :
const options = { flipHorizontal: true, // boolean value for if the video should be flipped, defaults to false maxContinuousChecks: Infinity // How many frames to go without running the bounding box detector. Defaults to infinity, but try a lower value if the detector is consistently producing bad predictions. // Threshold for discarding a prediction. Defaults to 0.8. // A threshold for removing multiple (likely duplicate) detections based on a "non-maximum suppression" algorithm. Defaults to 0.75 // A float representing the threshold for deciding whether boxes overlap too much in non-maximum suppression. Must be between [0, 1]. Defaults to 0.3. }
but HandPoints are detecting Outside the browser windows..

heres my try 👎
**
function setup() {
createCanvas(1500, 750);
video = createCapture(VIDEO);
video.size(1500, 950);
textSize(40);
textAlign(CENTER, CENTER);
var el = select('canvas').position(10, 30);
var canva = el.elt
canva.style.opacity = "0.8"
canva.style['pointer-events'] = "none"
var flipped ml5.flipImage(video)
handpose = ml5.handpose(flipped,options, modelReady);
handpose.on("predict", results => {
predictions = results;
});
video.hide();
}
function modelReady() {
console.log("Model ready!");
}
function draw() {
push();
image(flipped, 0, 0, 1500, 750);
pop()
drawKeypoints();
}
**

suggest me something
i have also tried
scale(-1,1) but nothing worked..
i m getting flipped video but not getting flipped HandPoints detected...Always Opposite to my hand position
WHAT SHOULD I DO ?

[tlsaeger]

Hi @nikkisingh111333 thanks for raising this and sorry for the late reply. Have you fixed this issue in the meantime. It seems like this is not only the case with the Handpose model, but also PoseNet which was mentioned here: #1198
We were pinged about this over on Discord and I will dig into this.

[shiffman]

This came up in my class this semester and relates to a larger, more complex discussion around the way PoseNet (and HandPose) works vs. other models that are typically used with video.

With PoseNet the video is "connected" behind the scenes and an event is triggered repeatedly. This is why you can't easily explicitly pass a modified version of the video.

// No mention of video! Happens when you load the model!
poseNet.on('pose', gotPose);

// Happens repeatedly without call to detect pose again!
function gotPose(results) {
  // do stuff
}

// Explicit designation of video
imageClassifier.classify(video, gotResults);

function gotResults(error, results) {
  // do stuff
  // call classify again to repeat the process!
  imageClassifier.classify(video, gotResults);
}

Students were confused by these different ways of working, as well as the PoseNet / HandPose event not having an error argument.

[sproutleaf]

Closing this issue for now and moving it to the new repo https://github.com/ml5js/ml5-next-gen, where we're working on the next iteration of ml5.js library.

Please see ml5js/ml5-next-gen#8.