3D reconstruction demo (Windows) - datumsPtr is nullptr in reconstruction3D post-process worker

[pichiste]

Issue summary

Hi there,
I'm attempting to get the 3D reconstruction demo working with input from some PS3 Eye cameras. I've got a custom producer (based on the WPointGrey class) handling the camera input and can see the pose detection working for each camera just fine. However, it seems the reconstruction3D worker is not receiving the pose data.

This is how the custom classes are initialized in openpose3d.cpp:

    // Frames producer (e.g. video, webcam, ...)
    //auto wPointGrey = std::make_shared<WPointGrey>();
    auto wPs3Eye = std::make_shared<WPs3Eye>(); // <---- ADDED
    // Processing
    auto wReconstruction3D = std::make_shared<WReconstruction3D>();
    // GUI (Display)
    auto wRender3D = std::make_shared<WRender3D>();

    op::Wrapper<std::vector<Datum3D>> opWrapper;
    // Add custom input
    const auto workerInputOnNewThread = true;
    //opWrapper.setWorkerInput(wPointGrey, workerInputOnNewThread);
    opWrapper.setWorkerInput(wPs3Eye, workerInputOnNewThread); // <---- ADDED
    // Add custom processing
    const auto workerProcessingOnNewThread = true;
    opWrapper.setWorkerPostProcessing(wReconstruction3D, workerProcessingOnNewThread);
    // Add custom output
    const auto workerOutputOnNewThread = true;
    opWrapper.setWorkerOutput(wRender3D, workerOutputOnNewThread);

I've only changed two lines, where I've swapped the WPointGrey producer for the custom WPs3Eye. As I mentioned above, 2D pose detection is working (it renders the webcam images + poses just fine!), but for some reason the pose data never shows up in the WReconstruction3D post-process.

More specifically, in reconstruction3D.cpp at the beginning of work function, datumsPtr is a nullptr:

void WReconstruction3D::work(std::shared_ptr<std::vector<Datum3D>>& datumsPtr)
{
    // User's post-processing (after OpenPose processing & before OpenPose outputs) here
        // datum.cvOutputData: rendered frame with pose or heatmaps
        // datum.poseKeypoints: Array<float> with the estimated pose
    try
    {
		if (datumsPtr == nullptr)
		{
			std::cout << "datumsPtr is nullptr!" << std::endl;
		}

        // Profiling speed
        const auto profilerKey = op::Profiler::timerInit(__LINE__, __FUNCTION__, __FILE__);
        if (datumsPtr != nullptr && /*!datumsPtr->empty() &&*/ datumsPtr->size() == 3)
        {

        ...

Any help with this would be much appreciated! If it's useful I can send along the webcam code, although it seems like it would be totally unrelated to the post-process stage (?).

Many thanks in advance!

Type of issue

• Help wanted

System configuration

Operating system: Windows 10 Pro
CUDA version: 8.0
cuDNN version: 5.1
GPU model: NVIDIA GeForce 1080 Ti
Caffe version: default
OpenCV version: default
Compiler: Visual Studio 2015

[gineshidalgo99]

Is WPs3Eye actually getting images? I.e. can you do some kind of imshow() or something similar in its work method to verify it is actually saving the image on the input cv::Mat of the Datum class?

[pichiste]

Yes, just tested and the images are being grabbed and put into Datum in WPs3Eye. For clarity here is the work method:

std::shared_ptr<std::vector<Datum3D>> WPs3Eye::workProducer()
{
	try
	{
		// Grab image from each camera
		const auto cvMats = acquireImages(cameraList, matList);

		// Images to userDatum
		auto datums3d = std::make_shared<std::vector<Datum3D>>(cvMats.size());
		for (auto i = 0u; i < cvMats.size(); i++)
		{
			datums3d->at(i).cvInputData = cvMats.at(i);
		}

		// TEST: Display images from camera 1 and 2
		if (!datums3d->empty())
		{
			cv::imshow("TEST CAM 1", datums3d->at(0).cvInputData);
			cv::imshow("TEST CAM 2", datums3d->at(1).cvInputData);
			cv::waitKey(0);
		}
		// Return Datum
		return datums3d;
	}
	catch (const std::exception& e)
	{
		this->stop();
		op::error(e.what(), __LINE__, __FUNCTION__, __FILE__);
		return nullptr;
	}
}

WRender3D is also displaying the camera images and rendering the 2D poses totally fine. Could it be something related to how I've set up the WReconstruction3D post-processing worker?

[gineshidalgo99]

No idea. Maybe the OpenGL visualization is not working on your system? Can you try imshow in the wRender3D and to try to output the keypoints with cout in both wRender3D and WReconstruction3D?

[pichiste]

In WReconstruction3D, I simply get a nullptr for the incoming datumPtr in the work method.

In WRender3D I'm able to display the images and print out the keypoints from the workConsumer method (image below). The OpenGL view itself works fine, but of course doesn't show the body pose as WReconstruction3D is never able to make the calculations since it's receiving no data.

[gineshidalgo99]

But wRender3D happens after WReconstruction3D. If WReconstruction3D has a nullptr, wRender3D will also have a nullptr.

And I am sorry, I have no idea what the error might actually be. As I do not have the multi-system camera installed anymore, I cannot debug it. Please, feel free to post the solution when you find it. Best

[pichiste]

Yeah, I also thought it was a bit weird that WReconstruction3D is getting a nullptr while WRender3D gets the data. I'll take another stab at it in the next couple of days and post what I find! Thanks

[pichiste]

Figured it out... it was a silly mistake on my part. In the WReconstruction3D work method, it checks for 3 cameras/images, but I'm only using 2, so it was never making it past the if statement:

if (datumsPtr != nullptr && /* !datumsPtr->empty() && */ datumsPtr->size() == 3)

The confusing part was that the work method receives a bunch of nullptr data between the real data, which is all I was seeing coming in. I'm not sure this is something I can change in WPs3Eye or if it's just the way it works, but either way it doesn't matter much.

Anyway, sorry for the bother! Thanks for your help!

[gineshidalgo99]

OK thanks for the feed-back!

[carstenschwede]

@pichiste Great that it works! Could you fork the repo with your added PS3Eye support or upload your modification somewhere? This would be awesome!

[gineshidalgo99]

@carstenschwede
The updated repo included this fix long time ago. The newest code already auto-adapts to the number of cameras, it doesn't required to hard code them anywhere.

EDITED: Sorry, I missed the part about PS3Eye support. No, that part is not incorporated, the incorporated part is to automatically adapt the code to the number of cameras. Sorry for the confusion. pichiste answered with his code.

[carstenschwede]

@gineshidalgo99 Sorry, must have missed it. Thanks for the info.

[pichiste]

Hi @carstenschwede, in case you just want the WPs3Eye producer, I've attached the files here (ps3Eye.hpp and ps3Eye.cpp). You should be able to use it the same as the WPointGrey producer, as stated in the first post.

I should note this is for Windows 10 + VS2015, and that WPs3Eye relies on these drivers:
https://github.com/inspirit/PS3EYEDriver

Here are my (old) notes on how to get the PS3EYEDriver drivers into the VS2015 project:

• Copy PS3EYEDriver source to project
• Copy libusb-1.0 library to 3rdparty\windows
• In project properites -> C/C++ -> Additional Include Directories, enter path to libusb-1.0 library (ex: ....\3rdparty\windows\libusb-1.0\include\libusb-1.0)
• In project properties -> Linker -> General -> Additional Library Directoires, enter path to libusb-1.0 dll directory (ex: ....\3rdparty\windows\libusb-1.0\MS64\dll
• In project properties -> Linker -> Input -> Additional Dependencies, enter: libusb-1.0.lib
• Copy libusb-1.0.dll to windows\x64\Release

I ended up putting this on the backburner before I fully got 3d reconstruction working, but I think it was just a matter of entering more accurate camera calibration parameters.

Would be curious to hear if you get this working!

ps3Eye_code.zip

[hadi1376tm]

 WRec

Hi, can i have the full code please?