Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

System.AccessViolationException throw if captures hand #15

Closed
takuya-takeuchi opened this issue Jun 12, 2020 · 33 comments
Closed

System.AccessViolationException throw if captures hand #15

takuya-takeuchi opened this issue Jun 12, 2020 · 33 comments
Assignees
@takuya-takeuchi
Labels
question Further information is requested

Comments

@takuya-takeuchi
Copy link
Owner

takuya-takeuchi commented Jun 12, 2020 •
edited
Loading

Report from @bruce987

@takuya-takeuchi

I was revisiting your site to see if there was any new issues and came across this. I've been working with OpenPoseDotNet for awhile now. I'm capturing live video feed of the hand and trying to determine direction. Anyway, if I move my hand into the webcam to adjust it I get an access violation every time. I have a secondary camera on a tripod. I'm just reporting this issue to let you know it does happen on a regular basis.

image

Originally posted by @bruce987 in #5 (comment)
@takuya-takeuchi takuya-takeuchi self-assigned this Jun 12, 2020
@takuya-takeuchi takuya-takeuchi mentioned this issue Jun 12, 2020
Open
@takuya-takeuchi
Copy link
Owner Author

@bruce987

  • What is Vector2?
    • If this is Unity, perhaps I can not support
  • Minimal code to reproduce issue if you can provide

@bruce987
Copy link

@takuya-takeuchi

Vector2 is part of Microsoft's System.Numerics. That class has Vector, Vector2, Vector3, and Vector4... and it is not part of Unity, though Unity does have its own Vector classes.

Here is the method that fails (RetrieveHandData). At this point I'm trying to get Hand movement to determine which direction it's being waved. The first argument is which hand and the second is the datum which is derived from the "StdSharedPtr" which you will see in the examples of OpenPose. It comes from the WUserOutput.cs.

    protected override void WorkConsumer(StdSharedPtr<Datum>[] datumsPtr)
    {
        try
        {
            // User's displaying/saving/other processing here
            // datum.cvOutputData: rendered frame with pose or heatmaps
            // datum.poseKeypoints: Array<float> with the estimated pose
            if (datumsPtr != null && datumsPtr.Length != 0)
            {
                var datum = datumsPtr[0].Get();

                var rightHandVector = RetrieveHandData(BodyPart.Right, datum);

                var leftHandVector = RetrieveHandData(BodyPart.Left, datum);

                ProcessData(rightHandVector, leftHandVector);

                DisplayResults(datum);
            }
        }
        catch (Exception e)
        {
            Stop();
            LogHelper.Error($"{MethodBase.GetCurrentMethod()?.Name}: {e.Message}");
        }
    }

    private static Vector2 RetrieveHandData(BodyPart whichSide, Datum datum)
    {
        var handVector = Vector2.Zero;

        try
        {
            switch (whichSide)
            {
                case BodyPart.Right:
                {
                    var handKeyPoints = datum.HandKeyPoints[1];

                    handVector = new Vector2(handKeyPoints[new[] {0, 10, 0}], // 10 = middleJoint2
                                             handKeyPoints[new[] {0, 10, 1}]);
                    break;
                }
                case BodyPart.Left:
                {
                    var handKeyPoints = datum.HandKeyPoints[0];

                    handVector = new Vector2(handKeyPoints[new[] {0, 10, 0}], // 10 = middleJoint2
                                             handKeyPoints[new[] {0, 10, 1}]);
                    break;
                }
                default:
                    handVector = default;
                    break;
            }
        }
        catch (Exception e)
        {
            LogHelper.Info($"{MethodBase.GetCurrentMethod()?.Name}: {whichSide} {e.Message}");
        }

        return handVector;
    }

@takuya-takeuchi
Copy link
Owner Author

@bruce987
One question.
Where does numer of middleJoint2 (10) come from?
I can not find this number from openpose example code.
openpose example does not show how to get hand key points,

@takuya-takeuchi takuya-takeuchi added the question Further information is requested label Jun 13, 2020
@bruce987
Copy link

@takuya-takeuchi

The number for middlejoint2 (10) comes from the OpenPose site itself. Scroll down toward the bottom to see the definition.

https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/doc/output.md

I had to dig around to find it. They explicitly define the Body25 parts, but not for the face or hand. The show the numeric number for face and hand, but do not actually give them names like Body25. So I assign them names. The previous post shows how I retrieved the hand data. (RetrieveHandData).

@takuya-takeuchi
Copy link
Owner Author

@bruce987
Wow!!!! Great!! Thanks a lot for your works!!

OK.
Let me check program.

@takuya-takeuchi
Copy link
Owner Author

@bruce987
I checked 07_HandFromImage.

image

It works fine.
Did you set true value to Flags.Hand?

This 07_HandFromImage uses this snipet.

// Required flags to enable heatmaps
Flags.Body = 0;
Flags.Hand = true;
Flags.HandDetector = 2;

But if this is changed to this, demo crashes due to above watch window value.

// Required flags to enable heatmaps
Flags.Body = 1;
Flags.Hand = false;
Flags.HandDetector = 0;

@takuya-takeuchi
Copy link
Owner Author

takuya-takeuchi commented Jun 13, 2020 •
edited
Loading

Name of parts for hand.
https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/41d8c087459fae844e477dda50a6f732e70f2cb8/src/openpose/pose/poseParameters.cpp#L182

But where is 0?

@bruce987
Copy link

Ok, let me check my code and look at example 7. I'm doing it a little different. This is my code that initializes the Wrapper.

    private static void ConfigureWrapper(Wrapper<Datum> opWrapper)
    {
        try
        {
            // Configuring OpenPose

            // logging_level
            OpenPose.CheckBool(0 <= Flags.LoggingLevel && Flags.LoggingLevel <= 255, "Wrong logging_level value.");
            ConfigureLog.PriorityThreshold = (Priority)Flags.LoggingLevel;
            Profiler.SetDefaultX((ulong)Flags.ProfileSpeed);
            
            // // For debugging
            // // Print all logging messages
            // ConfigureLog.PriorityThreshold = Priority.None;
            // // Print out speed values faster
            // Profiler.setDefaultX(100);

            // Applying user defined configuration - GFlags to program variables
            // producerType
            Flags.Camera = 1;
            var (producerType, producerString) = OpenPose.FlagsToProducer(Flags.ImageDir, Flags.Video, Flags.IpCamera, Flags.Camera, Flags.FlirCamera, Flags.FlirCameraIndex);

            // cameraSize
            var cameraSize = OpenPose.FlagsToPoint(Flags.CameraResolution, "-1x-1");
            
            // outputSize
            var outputSize = OpenPose.FlagsToPoint(Flags.OutputResolution, "-1x-1");
            
            // netInputSize
            var netInputSize = OpenPose.FlagsToPoint(Flags.NetResolution, "-1x-1");
            
            // faceNetInputSize
            var faceNetInputSize = OpenPose.FlagsToPoint(Flags.FaceNetResolution, "368x368 (multiples of 16)");
            
            // handNetInputSize
            var handNetInputSize = OpenPose.FlagsToPoint(Flags.HandNetResolution, "368x368 (multiples of 16)");
            
            // poseMode
            var poseMode = OpenPose.FlagsToPoseMode(Flags.Body);
            
            // poseModel
            var poseModel = OpenPose.FlagsToPoseModel(Flags.ModelPose);
            
            // JSON saving
            if (!string.IsNullOrEmpty(Flags.WriteKeyPoint))
            {
                LogHelper.Info("Flag `write_keypoint` is deprecated and will eventually be removed. Please, use `write_json` instead.", Priority.Max);
            }

            // key point Scale
            var keypointScale = OpenPose.FlagsToScaleMode(Flags.KeyPointScale);
            
            // heatmaps to add
            var heatMapTypes = OpenPose.FlagsToHeatMaps(Flags.HeatmapsAddParts, Flags.HeatmapsAddBackground, Flags.HeatmapsAddPAFs);
            var heatMapScale = OpenPose.FlagsToHeatMapScaleMode(Flags.HeatmapsScale);
            
            // >1 camera view?
            var multipleView = Flags.Enable3D || Flags.Views3D > 1 || Flags.FlirCamera;
            
            // Face and hand detectors
            var faceDetector = OpenPose.FlagsToDetector(Flags.FaceDetector);
            var handDetector = OpenPose.FlagsToDetector(Flags.HandDetector);
            
            // disable Google Logging
            const bool enableGoogleLogging = false;

            // Initializing the user custom classes
            // GUI (Display)
            var wUserOutput = new StdSharedPtr<UserWorkerConsumer<Datum>>(new WUserOutput());

            // Add custom processing
            opWrapper.SetWorker(WorkerType.Output, wUserOutput);

            // Pose configuration (use WrapperStructPose{} for default and recommended configuration)
            Flags.NumGpu = 1;
            //Flags.NumGpuStart = 1; // throws exception
            Flags.NumberPeopleMax = 1;
            Flags.FpsMax = 60;
            var pose = new WrapperStructPose(poseMode,
                                             netInputSize,
                                             outputSize,
                                             keypointScale,
                                             Flags.NumGpu,
                                             Flags.NumGpuStart,
                                             Flags.ScaleNumber,
                                             (float)Flags.ScaleGap,
                                             OpenPose.FlagsToRenderMode(Flags.RenderPose, multipleView),
                                             poseModel,
                                             !Flags.DisableBlending,
                                             (float)Flags.AlphaPose,
                                             (float)Flags.AlphaHeatmap,
                                             Flags.PartToShow,
                                             Flags.ModelFolder,
                                             heatMapTypes,
                                             heatMapScale,
                                             Flags.PartCandidates,
                                             (float)Flags.RenderThreshold,
                                             Flags.NumberPeopleMax,
                                             Flags.MaximizePositives,
                                             Flags.FpsMax,
                                             Flags.PrototxtPath,
                                             Flags.CaffeModelPath,
                                             (float)Flags.UpsamplingRatio,
                                             enableGoogleLogging);

            // Face configuration (use op::WrapperStructFace{} to disable it)
            //Flags.Face = true;
            var face = new WrapperStructFace(Flags.Face,
                                             faceDetector,
                                             faceNetInputSize,
                                             OpenPose.FlagsToRenderMode(Flags.FaceRender, multipleView, Flags.RenderPose),
                                             (float)Flags.FaceAlphaPose,
                                             (float)Flags.FaceAlphaHeatmap,
                                             (float)Flags.FaceRenderThreshold);

            // Hand configuration (use op::WrapperStructHand{} to disable it)
            Flags.Hand = true;
            var hand = new WrapperStructHand(Flags.Hand,
                                             handDetector,
                                             handNetInputSize,
                                             Flags.HandScaleNumber,
                                             (float) Flags.HandScaleRange,
                                             OpenPose.FlagsToRenderMode(Flags.HandRender, multipleView, Flags.RenderPose),
                                             (float) Flags.HandAlphaPose,
                                             (float) Flags.HandAlphaHeatmap,
                                             (float) Flags.HandRenderThreshold);

            // Extra functionality configuration (use op::WrapperStructExtra{} to disable it)
            //Flags.Enable3D = true; // throws exception
            var extra = new WrapperStructExtra(Flags.Enable3D,
                                               Flags.MinViews3D,
                                               Flags.Identification,
                                               Flags.Tracking,
                                               Flags.IkThreads);

            // Producer (use default to disable any input)
            Flags.ProcessRealTime = true;
            var input = new WrapperStructInput(producerType,
                                               producerString,
                                               Flags.FrameFirst,
                                               Flags.FrameStep,
                                               Flags.FrameLast,
                                               Flags.ProcessRealTime,
                                               Flags.FrameFlip,
                                               Flags.FrameRotate,
                                               Flags.FramesRepeat,
                                               cameraSize,
                                               Flags.CameraParameterPath,
                                               Flags.FrameUndistort,
                                               Flags.Views3D);

            // Output (comment or use default argument to disable any output)
            var output = new WrapperStructOutput(Flags.CliVerbose,
                                                 Flags.WriteKeyPoint,
                                                 OpenPose.StringToDataFormat(Flags.WriteKeyPointFormat),
                                                 Flags.WriteJson,
                                                 Flags.WriteCocoJson,
                                                 Flags.WriteCocoJsonVariants,
                                                 Flags.WriteCocoJsonVariant,
                                                 Flags.WriteImages,
                                                 Flags.WriteImagesFormat,
                                                 Flags.WriteVideo,
                                                 Flags.WriteVideoFps,
                                                 Flags.WriteVideoWithAudio,
                                                 Flags.WriteHeatmaps,
                                                 Flags.WriteHeatmapsFormat,
                                                 Flags.WriteVideoAdam,
                                                 Flags.WriteBvh,
                                                 Flags.UdpHost,
                                                 Flags.UdpPort);

            // No GUI. Equivalent to: opWrapper.configure(op::WrapperStructGui{});
            opWrapper.Configure(pose);
            opWrapper.Configure(face);
            opWrapper.Configure(hand);
            opWrapper.Configure(extra);
            opWrapper.Configure(input);
            opWrapper.Configure(output);

            // No GUI. Equivalent to: opWrapper.configure(op::WrapperStructGui{});
            // Set to single-thread (for sequential processing and/or debugging and/or reducing latency)
            Flags.DisableMultiThread = false;
            if (Flags.DisableMultiThread)
            {
                opWrapper.DisableMultiThreading();
            }
        }
        catch (Exception e)
        {
            LogHelper.Error($"{nameof(ConfigureWrapper)} {e.Message}");
        }
    }

@bruce987
Copy link

Keep in mind I'm capturing a video feed from a webcam and processing in real-time.

@takuya-takeuchi
Copy link
Owner Author

OK. But i can not find Flags.HandDetector. It could be 0 as default value.
It may be important.
Because OpenPose.FlagsToDetector(Flags.HandDetector); returns different value.

@takuya-takeuchi
Copy link
Owner Author

memo

XXXX = 0,
ThumbCarpometacarpal,
ThumbMetacarpophalangeal,
ThumbInterphalangeal,
ThumbFingerTip,
IndexMetacarpophalangeal,
IndexProximalInterphalangeal,
IndexDistalInterphalangeal,
IndexFingerTip,
MiddleMetacarpophalangeal,
MiddleProximalInterphalangeal,
MiddleDistalInterphalangeal,
MiddleFingerTip,
RingMetacarpophalangeal,
RingProximalInterphalangeal,
RingDistalInterphalangeal,
RingFingerTip,
PinkyMetacarpophalangeal,
PinkyProximalInterphalangeal,
PinkyDistalInterphalangeal,
PinkyFingerTip

@bruce987
Copy link

I will change the names next week. The value 10 still works for the middle finger joint.

I tried this...
Flags.HandDetector = 2;
var handDetector = OpenPose.FlagsToDetector(Flags.HandDetector);

This causes an early exception. So I put it back to just the FlagsToDetector without setting it to 2, then it works.

@bruce987
Copy link

If you hook up your webcam and run the app. While just looking at the camera, raise your hand so the hand is detected. Then move your hand toward the camera... the exception will occur.

@takuya-takeuchi
Copy link
Owner Author

I modified 11_AsynchronousCustomOutput.

        private static void ConfigureWrapper(Wrapper<Datum> opWrapper)
        {
            try
            {
+                Flags.Hand = true;
+                Flags.HandDetector = 0;

                // logging_level

image

It works fine while 1 minute running.
I tried

  • raise right hand
  • raise left hand
  • hidd right hand
  • hide left hand
  • raise both hands

But there is no problem

@bruce987
Copy link

bruce987 commented Jun 13, 2020 via email

@bruce987
Copy link

bruce987 commented Jun 13, 2020 via email

@bruce987
Copy link

bruce987 commented Jun 13, 2020 via email

@takuya-takeuchi
Copy link
Owner Author

Would like my project that does all this?

What does it mean?
Do you expect that I try the above all code, right? If so, no.

Did you try to move your hand to the webcam? I use a Logitech C922 on a tripod. I found the error when I moved my hand to adjust the camera.

Yes, my webcam Microsoft LifeCam Studio Q2F-00021 can not detect hand first time.
So I tried to move hand to camera and leave from camera to detect hand by openpose.

@takuya-takeuchi
Copy link
Owner Author

takuya-takeuchi commented Jun 13, 2020 •
edited
Loading

oh,... I forgot to insert new [] {0, 10, 0}

please wait!!

@bruce987
Copy link

bruce987 commented Jun 13, 2020 via email

@takuya-takeuchi
Copy link
Owner Author

@bruce987
OK, I was able to reproduce your issue.
Uhh.. it is weird.

@takuya-takeuchi
Copy link
Owner Author

App crash this scene.

image

@takuya-takeuchi
Copy link
Owner Author

takuya-takeuchi commented Jun 13, 2020 •
edited
Loading

@bruce987
I think this issue does not relate to distance between camera and hand.
Perhaps, it may occur when openpose does not detect any parts, face, hand or body.

After this, app crashes.

image

@bruce987
Copy link

Yes the app will run without issues for a very long time, but if you move your hand to the camera... EXCEPTION! I believe it may be happening in the C++ code.

I use a fixed size queue to store 20 data points. The queue is a ConcurrentQueue so its thread safe...

public class FixedSizedConcurrentQueue<TValue> : IProducerConsumerCollection<TValue>, IReadOnlyCollection<TValue>
{
    private readonly ConcurrentQueue<TValue> queue;

    private readonly object syncObject = new object();

    public int Size { get; }

    public bool IsEmpty => queue.IsEmpty;

    public int Count => queue.Count;

    bool ICollection.IsSynchronized => ((ICollection) queue).IsSynchronized;

    object ICollection.SyncRoot => ((ICollection) queue).SyncRoot;

    public FixedSizedConcurrentQueue(int size)
    {
        queue = new ConcurrentQueue<TValue>();
        Size = size;
    }

    public FixedSizedConcurrentQueue(int size, IEnumerable<TValue> collection)
    {
        queue = new ConcurrentQueue<TValue>(collection);
        Size = size;
    }

    // Not supported until .NET Standard 2.1
    public void Clear() => queue.Clear();

    public void CopyTo(TValue[] array, int index) => queue.CopyTo(array, index);

    void ICollection.CopyTo(Array array, int index) => ((ICollection) queue).CopyTo(array, index);

    public IEnumerator<TValue> GetEnumerator() => queue.GetEnumerator();

    IEnumerator IEnumerable.GetEnumerator() => ((IEnumerable<TValue>) this).GetEnumerator();

    public TValue[] ToArray() => queue.ToArray();

    public bool TryAdd(TValue item)
    {
        Enqueue(item);
        return true;
    }

    bool IProducerConsumerCollection<TValue>.TryTake(out TValue item) => TryDequeue(out item);

    public void Enqueue(TValue obj)
    {
        queue.Enqueue(obj);
        lock (syncObject)
        {
            while (queue.Count > Size)
            {
                queue.TryDequeue(out _);
            }
        }
    }

    public bool TryDequeue(out TValue result) => queue.TryDequeue(out result);

    public bool TryPeek(out TValue result) => queue.TryPeek(out result);
}

@takuya-takeuchi
Copy link
Owner Author

You can check array property.

Crash

image

Normal

image

@bruce987
Copy link

bruce987 commented Jun 13, 2020 via email

@bruce987
Copy link

bruce987 commented Jun 13, 2020 via email

@takuya-takeuchi
Copy link
Owner Author 

                var left = datum.HandKeyPoints[0];
                if (left.Volume != 0)
                {
                    //left.NumberDimensions
                    OpenPose.Log($"Left hand keypoints: {left[new[] { 0, 10, 0 }]}", Priority.High);
                }

It works fine!!

@bruce987
Copy link

bruce987 commented Jun 13, 2020 via email

@takuya-takeuchi
Copy link
Owner Author

OK.
Will you fix this issue on your company next week?
Please close this issue.

Thanks a lot for giving happy debugging time :)

@bruce987
Copy link

bruce987 commented Jun 13, 2020 via email

@bruce987
Copy link

@takuya-takeuchi

I'm not seeing the "Close Issue" at the bottom of the issue.

@bruce987
Copy link

@takuya-takeuchi

I think you own this thread. I can't close.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@takuya-takeuchi takuya-takeuchi

Labels
question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bruce987 @takuya-takeuchi